diff --git a/CHANGELOG.md b/CHANGELOG.md index 6516988db..c6e3b1899 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **Domain `Decision` + `Decision.from_algo`.** Copies `Episode.from_algo`: the caller supplies `owner_id` / `session_id` / `parent_id` (source memcell), and any algo-side `owner_id` or smuggled `parent_id` is dropped so one generic extract can fan out per user sender. No domain `Principle`. +- **`DecisionDailyFrontmatter`.** User-readable daily-log at `users//decisions/decision-.md` (`ENTRY_ID_PREFIX="dc"`), with `deprecated_entries` from day one. Not a dot-prefixed internal directory. +- **LanceDB `decision` table + `decision_repo`.** Daily-log chassis with dual BM25 (`decision_tokens` / `reason_tokens`), nullable vector, and `deprecated_by` from day one. Wired into `_BUSINESS_SCHEMAS`, `BUSINESS_SCHEMAS_WITH_VECTOR`, and cascade `_TABLE_SPECS` (embed text = `r["decision"]`). +- **`DecisionWriter` / `DecisionReader`.** Append-only daily-log at `users//decisions/decision-.md` with `dc_` entry ids. Markdown is the SoT; cascade projects into Lance. +- **`extract_decision` OME strategy (`enabled=True`) + `DecisionExtracted` (`source=pipeline`).** One `DecisionExtractor` call per memcell (no `sender_id`); EverOS fans out one daily-log copy per user sender and emits `DecisionExtracted` after each write. An empty list is success. +- **`DecisionHandler` + `KIND_REGISTRY` kind `decision`.** Markdown → Lance projection; embed only the Decision body (soft dependency); dual BM25 `decision_tokens` / `reason_tokens`. `deprecated_by` still from chassis `deprecated_entries`. +- **Search `data.decisions` + Get `memory_type=decision`.** User-partition Decision recall (`DecisionRecaller`: dual BM25 + cosine) fused with `rrf` (never `arank`). `kinds` filters the episode / decision lanes (`principle` is not a kind). Get lists Decision rows with `deprecated_by IS NULL`; there is no `GetMemoryType.PRINCIPLE`. +- **`trigger_decision_clustering` OME strategy.** On `DecisionExtracted(source=pipeline)` embeds `decision_text` and geometry-merges into sqlite clusters with `kind=decision` / `member_type=decision` (not the `user_memory` episode track). Emits `DecisionClusterUpdated` with a row snapshot. Cascade Phase 2 backfill synthesizes the same events for existing Lance decision rows (`parent_type=memcell`). Embed missing → debug no-op. +- **`principles.md` + Lance `principle` + `PrincipleHandler`.** Single-file rewrite via existing `ProfileWriter` (`users//principles.md`, `type=principle`). Cascade explodes the frontmatter list into N KV rows (`id=_`). No vector, no BM25, not in `_TABLE_SPECS` / `BUSINESS_SCHEMAS_WITH_VECTOR`. `KIND_REGISTRY` name `principle` is the cascade projection only — not a product Kind. +- **`extract_principles` + Search `include_principles`.** On `DecisionClusterUpdated`, unions every sqlite `kind=decision` cluster (one `PrincipleExtractor` call per cluster) into one `principles.md`. Search attaches the Lance KV rows when `include_principles=true` (`data.principles` always present). Not a kind: `kinds: ["principle"]` stays 422; agent owners ignore the flag. No Get `memory_type=principle`. +- **`DecisionReflectionOrchestrator` + `reflect_decisions` Cron (`enabled=false`).** Select → Merge (`DecisionReflector.areflect` → Decision DTO) → markdown `parent_type=cluster` → `DecisionExtracted(source="reflection")` (no wait; clustering is pipeline-only) → deprecate md `deprecated_entries` + Lance `deprecated_by`. No atomic-fact path. Sibling of episode reflection — does not edit `ReflectionOrchestrator`. +- **Decision closed-loop integration (FakeLLM).** `extract_decision` → `users//decisions/decision-*.md` → Cascade `decision` row → keyword Search recalls 「设备 Runtime 为什么使用 Rust？」 in `data.decisions`. No real model. `kinds: ["principle"]` / Get `memory_type=principle` stay 422. + +### Changed + +- **`everalgo-user-memory` 0.4.0 → 0.8.0** (path-pin to the sibling EverAlgo checkout until 0.8.0 is on PyPI) and **`everalgo-agent-memory` 0.4.0 → 0.5.0**. 0.8.0 is the first user-memory release that exports `Decision` / `Principle` types and extractors. `extract_decision` calls `DecisionExtractor` on each `UserPipelineStarted`. `AlgoEpisode` construction now passes the required `summary` field introduced in everalgo-core 0.5.0. + ## [1.2.3] - 2026-08-07 **Background maintenance that fails loudly instead of quietly.** A soak run on diff --git a/docs/api.md b/docs/api.md index 422b47ffa..559409c78 100644 --- a/docs/api.md +++ b/docs/api.md @@ -476,14 +476,16 @@ should be reserved for offline or background workflows. | Value | Track | Returned in `data.` | |---|---|---| | `"episode"` | user | `data.episodes` — [GetEpisodeItem](#getepisodeitem) | +| `"decision"` | user | `data.decisions` — [GetDecisionItem](#getdecisionitem) | | `"profile"` | user | `data.profiles` — [GetProfileItem](#getprofileitem) | | `"agent_case"` | agent | `data.agent_cases` — [GetAgentCaseItem](#getagentcaseitem) | | `"agent_skill"` | agent | `data.agent_skills` — [GetAgentSkillItem](#getagentskillitem) | `memory_type` must match the requested owner kind: `"episode"` / -`"profile"` require `user_id`; `"agent_case"` / `"agent_skill"` -require `agent_id`. The mismatching combinations are rejected with -`422`. +`"decision"` / `"profile"` require `user_id`; `"agent_case"` / +`"agent_skill"` require `agent_id`. The mismatching combinations are +rejected with `422`. Principle is Meta Memory — it is not a +`memory_type` value. ## Endpoints @@ -657,12 +659,13 @@ optional final LLM rerank. Returns ranked items grouped by kind. | `radius` | `number \| null` | no | `null` | `0.0 ≤ x ≤ 1.0` if set | | `min_score` | `number \| null` | no | `null` | `0.0 ≤ x ≤ 1.0` if set | | `include_profile` | `boolean` | no | `false` | — | +| `kinds` | `array<"episode" \| "decision"> \| null` | no | `null` | user-partition only; empty list is `422` | | `enable_llm_rerank` | `boolean` | no | `false` | — | | `filters` | [FilterNode](#filternode-filter-dsl) `\| null` | no | `null` | — | **`user_id` / `agent_id`** — **Exactly one** must be set. Determines which track is searched: `user_id` → user-memory (episodes / -profiles); `agent_id` → agent-memory (cases / skills). +decisions / profiles); `agent_id` → agent-memory (cases / skills). **`app_id` / `project_id`** — Scope identifiers; results never cross scopes. @@ -702,11 +705,20 @@ independent of `radius` (which is a per-recall cosine threshold). **`include_profile`** — When `user_id` is set, also fetch the user's profile and include it in `data.profiles`. The profile is not ranked; `score` is `null`. Ignored when `agent_id` is set. +Independent of `kinds`. + +**`kinds`** — Optional user-partition kind filter. `null` (default) +searches episode and decision in parallel. `["decision"]` / +`["episode"]` restrict the lanes; `["episode", "decision"]` is the +same as omitting the field. Rejected when `agent_id` is set, when +the list is empty, or when a value other than `"episode"` / +`"decision"` is supplied (`"principle"` is not a kind). **`enable_llm_rerank`** — Opt-in LLM rerank pass for `method: "hybrid"`. Applies to `agent_case` and `agent_skill` fusion only; the episode hybrid path has built-in fact eviction and -ignores this flag. Adds one LLM call per request. Ignored by +ignores this flag. Decision HYBRID fuses with RRF and also ignores +it. Adds one LLM call per request. Ignored by `keyword` / `vector` (no fusion to rerank) and `agentic` (uses its own cross-encoder loop). @@ -716,14 +728,15 @@ it does not perturb the ranker. #### Response body -`200 OK` returns a SuccessEnvelope wrapping `SearchData`. All five +`200 OK` returns a SuccessEnvelope wrapping `SearchData`. All kind arrays are always present so client code can iterate without branching on owner type; arrays that do not apply to the requested owner kind stay as `[]`. | Field | Type | Notes | |---|---|---| -| `episodes` | `array` | Populated when `user_id` is set | +| `episodes` | `array` | Populated when `user_id` is set (unless `kinds` excludes `"episode"`) | +| `decisions` | `array` | Populated when `user_id` is set (unless `kinds` excludes `"decision"`). HYBRID fuses BM25 + vector with RRF (not `arank`) | | `profiles` | `array` | Populated when `user_id` is set **and** `include_profile=true` | | `agent_cases` | `array` | Populated when `agent_id` is set | | `agent_skills` | `array` | Populated when `agent_id` is set | @@ -751,6 +764,27 @@ query within this episode (already nested, no separate call needed). | `score` | `number` | Fused retrieval score for this episode | | `atomic_facts` | `array` | Sub-facts extracted from the same episode that matched the query | +#### SearchDecisionItem + +User-track decision hit. `score` is the fused retrieval score (RRF +on hybrid; raw BM25 / cosine on keyword / vector). There is no +`sender_ids` field — tags are labels, not conversation participants. + +| Field | Type | Notes | +|---|---|---| +| `id` | `string` | `_dc__` | +| `user_id` | `string \| null` | Owner of this decision | +| `app_id` | `string` | Scope where the decision lives | +| `project_id` | `string` | Scope where the decision lives | +| `session_id` | `string \| null` | Originating session; `null` when the extract did not bind one | +| `timestamp` | `string` | ISO-8601 with timezone offset — see [Conventions](#conventions) | +| `title` | `string` | Short title of the trade-off | +| `decision` | `string` | Decision body (the retrieval / embed anchor) | +| `reason` | `string` | Why the trade-off was made | +| `impact` | `string \| null` | Optional downstream effect | +| `tags` | `array` | Caller-supplied labels | +| `score` | `number` | Fused retrieval score for this decision | + #### SearchAtomicFactItem A single-sentence fact pulled out of an episode during extraction. @@ -880,6 +914,7 @@ Response (real capture): ] } ], + "decisions": [], "profiles": [], "agent_cases": [], "agent_skills": [], @@ -910,15 +945,16 @@ for UI browsing, exports, or filtered scans. | `filters` | [FilterNode](#filternode-filter-dsl) `\| null` | no | `null` | — | **`user_id` / `agent_id`** — **Exactly one** must be set, and it must -match the track implied by `memory_type` (`"episode"` / `"profile"` -require `user_id`; `"agent_case"` / `"agent_skill"` require +match the track implied by `memory_type` (`"episode"` / `"decision"` / +`"profile"` require `user_id`; `"agent_case"` / `"agent_skill"` require `agent_id`). **`app_id` / `project_id`** — Scope identifiers. **`memory_type`** — Which item kind to list; see [GetMemoryType](#getmemorytype). The route populates exactly one of -the four arrays in `data` based on this value. +the kind arrays in `data` based on this value. `"principle"` is not +a valid value. **`page`** — 1-indexed page number. Together with `page_size` determines the window. The response's `total_count` reports how many @@ -938,13 +974,14 @@ predicate-based filtering before pagination. #### Response body -`200 OK` returns a SuccessEnvelope wrapping `GetData`. The four +`200 OK` returns a SuccessEnvelope wrapping `GetData`. The kind arrays are always present so client code can iterate without branching on `memory_type`; exactly one is populated. | Field | Type | Notes | |---|---|---| | `episodes` | `array` | Populated when `memory_type="episode"` | +| `decisions` | `array` | Populated when `memory_type="decision"` | | `profiles` | `array` | Populated when `memory_type="profile"` | | `agent_cases` | `array` | Populated when `memory_type="agent_case"` | | `agent_skills` | `array` | Populated when `memory_type="agent_skill"` | @@ -971,6 +1008,25 @@ sub-facts). | `episode` | `string` | Full extracted narrative | | `type` | `"Conversation"` | — | +#### GetDecisionItem + +Same shape as [SearchDecisionItem](#searchdecisionitem) **minus** +`score` (listing is unranked). + +| Field | Type | Notes | +|---|---|---| +| `id` | `string` | `_dc__` | +| `user_id` | `string \| null` | Owner | +| `app_id` | `string` | Scope | +| `project_id` | `string` | Scope | +| `session_id` | `string \| null` | Originating session; `null` when unbound | +| `timestamp` | `string` | ISO-8601 with timezone offset — see [Conventions](#conventions) | +| `title` | `string` | Short title of the trade-off | +| `decision` | `string` | Decision body | +| `reason` | `string` | Why the trade-off was made | +| `impact` | `string \| null` | Optional downstream effect | +| `tags` | `array` | Caller-supplied labels | + #### GetProfileItem | Field | Type | Notes | @@ -1051,6 +1107,7 @@ Response (real capture): "type": "Conversation" } ], + "decisions": [], "profiles": [], "agent_cases": [], "agent_skills": [], diff --git a/docs/architecture.md b/docs/architecture.md index afb5efe30..9f020d844 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -217,7 +217,7 @@ optional `everalgo-parser` extra), imported under the `everalgo` namespace, holding **only memory extraction algorithms**: - `everalgo.parser` — multi-modal parsing (optional `[multimodal]` extra) -- `everalgo.user_memory` — ConvMemCell / Episode / Foresight / AtomicFact / Profile extractors +- `everalgo.user_memory` — ConvMemCell / Episode / Foresight / AtomicFact / Profile / Decision extractors - `everalgo.agent_memory` — AgentMemCell / Case / Skill extractors - `everalgo.rank` — boundary detection / fusion + rerank - `everalgo.knowledge` — KnowledgeExtractor (document parse + topic extraction) diff --git a/docs/openapi.json b/docs/openapi.json index b55ba73ae..007767780 100644 --- a/docs/openapi.json +++ b/docs/openapi.json @@ -2484,6 +2484,13 @@ "type": "array", "title": "Episodes" }, + "decisions": { + "items": { + "$ref": "#/components/schemas/GetDecisionItem" + }, + "type": "array", + "title": "Decisions" + }, "profiles": { "items": { "$ref": "#/components/schemas/GetProfileItem" @@ -2519,7 +2526,94 @@ "additionalProperties": false, "type": "object", "title": "GetData", - "description": "Body of ``response.data``.\n\nAll four arrays are always present so client code can iterate\nwithout branching on ``memory_type``; the route populates exactly\none." + "description": "Body of ``response.data``.\n\nAll kind arrays are always present so client code can iterate\nwithout branching on ``memory_type``; the route populates exactly\none." + }, + "GetDecisionItem": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "User Id" + }, + "app_id": { + "type": "string", + "title": "App Id", + "default": "default" + }, + "project_id": { + "type": "string", + "title": "Project Id", + "default": "default" + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id" + }, + "timestamp": { + "type": "string", + "format": "date-time", + "title": "Timestamp" + }, + "title": { + "type": "string", + "title": "Title" + }, + "decision": { + "type": "string", + "title": "Decision" + }, + "reason": { + "type": "string", + "title": "Reason" + }, + "impact": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Impact" + }, + "tags": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Tags" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id", + "user_id", + "timestamp", + "title", + "decision", + "reason" + ], + "title": "GetDecisionItem", + "description": "Decision listing item — always user-scoped. No score (unranked)." }, "GetEpisodeItem": { "properties": { @@ -2601,12 +2695,13 @@ "type": "string", "enum": [ "episode", + "decision", "profile", "agent_case", "agent_skill" ], "title": "GetMemoryType", - "description": "The four kinds enumerated by ``/get``.\n\n``episode`` and ``profile`` are user-owned; ``agent_case`` and\n``agent_skill`` are agent-owned. Cross-pairs are rejected by\n:meth:`GetRequest._validate_owner_memory_type_pair`.\n\nNaming note: all four values use the bare kind name (no\n``_memory`` suffix) and match the LanceDB table name + everalgo\ntype name for that kind." + "description": "The kinds enumerated by ``/get``.\n\n``episode``, ``decision``, and ``profile`` are user-owned;\n``agent_case`` and ``agent_skill`` are agent-owned. Cross-pairs\nare rejected by :meth:`GetRequest._validate_owner_memory_type_pair`.\nThere is no ``principle`` value — Principle is Meta Memory and is\nnot listed here.\n\nNaming note: values use the bare kind name (no ``_memory`` suffix)\nand match the LanceDB table name for that kind." }, "GetProfileItem": { "properties": { @@ -3298,6 +3393,13 @@ "type": "array", "title": "Episodes" }, + "decisions": { + "items": { + "$ref": "#/components/schemas/SearchDecisionItem" + }, + "type": "array", + "title": "Decisions" + }, "profiles": { "items": { "$ref": "#/components/schemas/SearchProfileItem" @@ -3305,6 +3407,13 @@ "type": "array", "title": "Profiles" }, + "principles": { + "items": { + "$ref": "#/components/schemas/SearchPrincipleItem" + }, + "type": "array", + "title": "Principles" + }, "agent_cases": { "items": { "$ref": "#/components/schemas/SearchAgentCaseItem" @@ -3330,7 +3439,99 @@ "additionalProperties": false, "type": "object", "title": "SearchData", - "description": "Body of ``response.data``.\n\nAll five arrays are always present so client code can iterate without\nbranching on ``owner_type``. Routes not applicable to the request's\nowner type stay as ``[]``. ``unprocessed_messages`` is filled only\nwhen ``filters.session_id`` is present as a top-level eq scalar —\nin-flight buffer rows are scope-tagged but unattributed (no\n``user_id``), so session is the only meaningful query dimension." + "description": "Body of ``response.data``.\n\nAll arrays are always present so client code can iterate without\nbranching on ``owner_type``. Routes not applicable to the request's\nowner type stay as ``[]``. ``unprocessed_messages`` is filled only\nwhen ``filters.session_id`` is present as a top-level eq scalar —\nin-flight buffer rows are scope-tagged but unattributed (no\n``user_id``), so session is the only meaningful query dimension." + }, + "SearchDecisionItem": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "User Id" + }, + "app_id": { + "type": "string", + "title": "App Id", + "default": "default" + }, + "project_id": { + "type": "string", + "title": "Project Id", + "default": "default" + }, + "session_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Session Id" + }, + "timestamp": { + "type": "string", + "format": "date-time", + "title": "Timestamp" + }, + "title": { + "type": "string", + "title": "Title" + }, + "decision": { + "type": "string", + "title": "Decision" + }, + "reason": { + "type": "string", + "title": "Reason" + }, + "impact": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Impact" + }, + "tags": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Tags" + }, + "score": { + "type": "number", + "title": "Score" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id", + "user_id", + "timestamp", + "title", + "decision", + "reason", + "score" + ], + "title": "SearchDecisionItem", + "description": "Decision hit — always user-scoped. Instance Kind, HYBRID recall." }, "SearchEpisodeItem": { "properties": { @@ -3514,6 +3715,77 @@ "title": "SearchMethod", "description": "Public method enum. RRF / LR / vector_anchored are hidden under HYBRID." }, + "SearchPrincipleItem": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "user_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "User Id" + }, + "app_id": { + "type": "string", + "title": "App Id", + "default": "default" + }, + "project_id": { + "type": "string", + "title": "Project Id", + "default": "default" + }, + "title": { + "type": "string", + "title": "Title" + }, + "statement": { + "type": "string", + "title": "Statement" + }, + "source_entry_ids": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Source Entry Ids" + }, + "timestamp": { + "type": "string", + "format": "date-time", + "title": "Timestamp" + }, + "score": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Score" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "id", + "user_id", + "title", + "statement", + "timestamp" + ], + "title": "SearchPrincipleItem", + "description": "One engineering principle — KV fetch, not a ranked kind.\n\n``score`` is always ``None`` (no query-relevance). ``id`` is the\nLance PK ``_``. Agent owners never receive\nthis array." + }, "SearchProfileItem": { "properties": { "id": { @@ -3649,6 +3921,29 @@ "title": "Include Profile", "default": false }, + "include_principles": { + "type": "boolean", + "title": "Include Principles", + "default": false + }, + "kinds": { + "anyOf": [ + { + "items": { + "type": "string", + "enum": [ + "episode", + "decision" + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Kinds" + }, "enable_llm_rerank": { "type": "boolean", "title": "Enable Llm Rerank", diff --git a/pyproject.toml b/pyproject.toml index 991d9c91b..61d8aea9a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -76,9 +76,10 @@ dependencies = [ "watchfiles>=0.21.0", # native fs watcher for config hot reload "anyio>=4.0", # Async file I/O (anyio.Path, to_thread.run_sync) for the markdown layer - # Algorithm library (everalgo monorepo, published on PyPI). - "everalgo-user-memory==0.4.0", - "everalgo-agent-memory==0.4.0", + # Algorithm library. user-memory 0.8.0 (Decision) is not on PyPI yet; + # [tool.uv.sources] path-pins core + user-memory to the sibling EverAlgo checkout. + "everalgo-user-memory==0.8.0", + "everalgo-agent-memory==0.5.0", "everalgo-rank==0.4.1", "everalgo-knowledge==0.1.1", ] @@ -102,6 +103,12 @@ Changelog = "https://github.com/EverMind-AI/EverOS/blob/main/CHANGELOG.md" requires = ["hatchling"] build-backend = "hatchling.build" +# Workspace-local until everalgo-core 0.6.0 and everalgo-user-memory 0.8.0 are on PyPI. +# Paths assume EverOS and EverAlgo are sibling checkouts. +[tool.uv.sources] +everalgo-core = { path = "../EverAlgo/packages/everalgo-core", editable = true } +everalgo-user-memory = { path = "../EverAlgo/packages/everalgo-user-memory", editable = true } + [tool.hatch.build.targets.wheel] packages = ["src/everos"] diff --git a/scripts/check_consistency.py b/scripts/check_consistency.py index 80923a330..92cc4eac4 100755 --- a/scripts/check_consistency.py +++ b/scripts/check_consistency.py @@ -17,8 +17,8 @@ --mode readonly Bypass the lifespan stack, open LanceDB with a fresh read connection, read md directly. Safe even on an active corpus, but only - covers the three daily-log kinds (episode / - atomic_fact / foresight). + covers the four daily-log kinds (episode / + atomic_fact / foresight / decision). Examples: scripts/check_consistency.py ~/.everos-locomo-all-kv-fast @@ -84,7 +84,13 @@ async def _scan_monotonicity(corpus: Path) -> list[MonotonicityReport]: """Walk all daily-log md files; report id-counter monotonicity per file.""" from everos.core.persistence import MarkdownReader - daily_dirs = ("/episodes/", "/.atomic_facts/", "/.foresights/", "/.agent_cases/") + daily_dirs = ( + "/episodes/", + "/.atomic_facts/", + "/.foresights/", + "/decisions/", + "/.agent_cases/", + ) reports: list[MonotonicityReport] = [] for md in sorted(corpus.rglob("*.md")): rel = md.relative_to(corpus).as_posix() @@ -208,14 +214,16 @@ async def run_lifespan_mode(corpus: Path) -> int: async def run_readonly_mode(corpus: Path, owners_filter: list[str] | None) -> int: """Direct LanceDB read + md read; no lifespan / cascade / ome started. - Covers the three daily-log kinds; agent_case + user_profile + agent_skill - are NOT checked in this mode (use --mode lifespan on an idle corpus - snapshot for full coverage). + Covers the four daily-log kinds (episode / atomic_fact / foresight / + decision); agent_case + user_profile + agent_skill are NOT checked + in this mode (use --mode lifespan on an idle corpus snapshot for + full coverage). """ import lancedb from everos.core.persistence import MarkdownReader from everos.memory.cascade.handlers.atomic_fact import AtomicFactHandler + from everos.memory.cascade.handlers.decision import DecisionHandler from everos.memory.cascade.handlers.episode import EpisodeHandler from everos.memory.cascade.handlers.foresight import ForesightHandler from tests._consistency_assertions import _daily_log_sha_for_entry @@ -226,6 +234,7 @@ async def run_readonly_mode(corpus: Path, owners_filter: list[str] | None) -> in ("episode", "episodes", "episode-", EpisodeHandler), ("atomic_fact", ".atomic_facts", "atomic_fact-", AtomicFactHandler), ("foresight", ".foresights", "foresight-", ForesightHandler), + ("decision", "decisions", "decision-", DecisionHandler), ] # Pick owners diff --git a/src/everos/config/default_ome.toml b/src/everos/config/default_ome.toml index f20de373a..6f0328c2b 100644 --- a/src/everos/config/default_ome.toml +++ b/src/everos/config/default_ome.toml @@ -38,15 +38,33 @@ # [strategies.extract_foresight] # enabled = true +# Decision extraction (runs per memcell). One LLM call for the whole +# slice, then one md copy per user sender. Enabled by default (unlike +# extract_foresight) so Search can consume decisions once that route +# lands. Uncomment only to turn it off for evaluation runs: +# [strategies.extract_decision] +# enabled = false + # Profile clustering trigger (fires on each EpisodeExtracted event). # [strategies.trigger_profile_clustering] # enabled = true +# Decision clustering trigger (fires on each DecisionExtracted event). +# Sqlite kind=decision (not user_memory). Uncomment only to turn it off: +# [strategies.trigger_decision_clustering] +# enabled = false + # User-profile extraction (runs after clustering trigger fires). Set # enabled = false to skip in evaluation runs. # [strategies.extract_user_profile] # enabled = true +# Principle extraction (runs after decision clustering). Unions every +# kind=decision sqlite cluster into one principles.md per user. LLM, +# not embed — registered in ALWAYS. Uncomment only to turn it off: +# [strategies.extract_principles] +# enabled = false + # ── Reflection ────────────────────────────────────────────────────────── # Offline memory consolidation. Disabled by default. To enable, @@ -56,6 +74,12 @@ # enabled = false # cron = "0 2 * * 1" +# Decision reflection. Same cadence, independently default-off. Merges +# kind=decision sqlite clusters into one Decision (not a Principle). +# [strategies.reflect_decisions] +# enabled = false +# cron = "0 2 * * 1" + # ── Agent-memory pipeline ─────────────────────────────────────────────── # Agent case extraction (runs per agent memcell). One per tool call cycle. diff --git a/src/everos/config/prompt_slots/decision_extract.yaml b/src/everos/config/prompt_slots/decision_extract.yaml new file mode 100644 index 000000000..8d2d26cd9 --- /dev/null +++ b/src/everos/config/prompt_slots/decision_extract.yaml @@ -0,0 +1,21 @@ +# Custom prompt slot for DecisionExtractor.aextract. +# +# Default behaviour +# Leave this slot disabled (``enabled: false``). The strategy will pass +# ``prompt=None`` through to algo, which falls back to the everalgo +# bundled default prompt — see: +# everalgo/user_memory/prompts/en/decision.py +# (the strategy calls ``aextract`` with no ``sender_id``; the +# whole-memcell ``DECISION_GENERATION_PROMPT`` is the only path) +# +# To customise +# 1. Read the algo default at the path above; note the required +# placeholders ``{CONVERSATION_TEXT}`` and ``{language_rule}``. +# 2. Replace the ``template`` body below with your prompt. +# 3. Flip ``enabled`` to ``true``. +# +# When ``enabled: false`` or ``template`` is empty, the strategy sends +# ``prompt=None`` and the algo default is used (zero override cost). + +enabled: false +template: "" diff --git a/src/everos/entrypoints/cli/commands/_backfill_cmd.py b/src/everos/entrypoints/cli/commands/_backfill_cmd.py index cf7d3bf08..58520a051 100644 --- a/src/everos/entrypoints/cli/commands/_backfill_cmd.py +++ b/src/everos/entrypoints/cli/commands/_backfill_cmd.py @@ -183,9 +183,12 @@ def _print_vectors_estimate(total_rows: int, total_tokens: int) -> None: ) -def _print_clusters_estimate(episode_count: int, case_count: int) -> None: +def _print_clusters_estimate( + episode_count: int, case_count: int, *, decision_count: int = 0 +) -> None: typer.echo(f" episodes to cluster: {episode_count:,}") - typer.echo(f" agent cases to cluster: {case_count:,}\n") + typer.echo(f" agent cases to cluster: {case_count:,}") + typer.echo(f" decisions to cluster: {decision_count:,}\n") typer.echo( " Uses your embedding provider (and LLM for agent-case merges) — " "cost depends on provider pricing.\n" @@ -268,8 +271,10 @@ def server_running(self) -> None: def estimate_vectors(self, rows: int, tokens: int) -> None: _print_vectors_estimate(rows, tokens) - def estimate_clusters(self, episodes: int, cases: int) -> None: - _print_clusters_estimate(episodes, cases) + def estimate_clusters( + self, episodes: int, cases: int, *, decisions: int = 0 + ) -> None: + _print_clusters_estimate(episodes, cases, decision_count=decisions) def estimate_skills(self, cases: int, clusters: int) -> None: _print_skills_estimate(cases, clusters) diff --git a/src/everos/infra/persistence/lancedb/__init__.py b/src/everos/infra/persistence/lancedb/__init__.py index 78ea873ed..3419cbc39 100644 --- a/src/everos/infra/persistence/lancedb/__init__.py +++ b/src/everos/infra/persistence/lancedb/__init__.py @@ -12,11 +12,11 @@ from everos.infra.persistence.lancedb import ( get_connection, get_table, dispose_connection, - Episode, AtomicFact, Foresight, AgentCase, AgentSkill, UserProfile, - KnowledgeTopic, - episode_repo, atomic_fact_repo, foresight_repo, + Episode, AtomicFact, Decision, Foresight, AgentCase, AgentSkill, + UserProfile, Principle, KnowledgeTopic, + episode_repo, atomic_fact_repo, decision_repo, foresight_repo, agent_case_repo, agent_skill_repo, user_profile_repo, - knowledge_topic_repo, + principle_repo, knowledge_topic_repo, ) Three index kinds: scalar / BM25 / vector. Tables are created lazily on @@ -40,26 +40,32 @@ from .repos import agent_case_repo as agent_case_repo from .repos import agent_skill_repo as agent_skill_repo from .repos import atomic_fact_repo as atomic_fact_repo +from .repos import decision_repo as decision_repo from .repos import episode_repo as episode_repo from .repos import foresight_repo as foresight_repo from .repos import knowledge_topic_repo as knowledge_topic_repo +from .repos import principle_repo as principle_repo from .repos import user_profile_repo as user_profile_repo from .tables import AgentCase as AgentCase from .tables import AgentSkill as AgentSkill from .tables import AtomicFact as AtomicFact +from .tables import Decision as Decision from .tables import Episode as Episode from .tables import Foresight as Foresight from .tables import KnowledgeTopic as KnowledgeTopic from .tables import ParentType as ParentType +from .tables import Principle as Principle from .tables import UserProfile as UserProfile _BUSINESS_SCHEMAS = ( Episode, AtomicFact, + Decision, Foresight, AgentCase, AgentSkill, UserProfile, + Principle, KnowledgeTopic, ) @@ -173,6 +179,7 @@ async def migrate_fts_indexes() -> None: BUSINESS_SCHEMAS_WITH_VECTOR: tuple[type[BaseLanceTable], ...] = ( Episode, AtomicFact, + Decision, Foresight, AgentCase, AgentSkill, @@ -180,8 +187,9 @@ async def migrate_fts_indexes() -> None: ) """Business schemas whose ``vector`` column needs the v2 nullability migration. ``Episode.subject_vector`` was already nullable in v1.1.1 and -is intentionally left out of this migration. ``UserProfile`` has no -vector column; ``knowledge_document`` has no LanceDB table.""" +is intentionally left out of this migration. ``UserProfile`` and +``Principle`` have no vector column; ``knowledge_document`` has no +LanceDB table.""" async def migrate_table_schemas() -> None: @@ -371,16 +379,19 @@ async def drop_business_tables() -> list[str]: "AgentCase", "AgentSkill", "AtomicFact", + "Decision", "Episode", "Foresight", "KnowledgeTopic", "LanceDBMigrationError", "LanceDBSchemaMismatchError", "ParentType", + "Principle", "UserProfile", "agent_case_repo", "agent_skill_repo", "atomic_fact_repo", + "decision_repo", "dispose_connection", "drop_business_tables", "ensure_business_indexes", @@ -391,6 +402,7 @@ async def drop_business_tables() -> list[str]: "knowledge_topic_repo", "migrate_fts_indexes", "migrate_table_schemas", + "principle_repo", "user_profile_repo", "verify_business_schemas", ] diff --git a/src/everos/infra/persistence/lancedb/repos/__init__.py b/src/everos/infra/persistence/lancedb/repos/__init__.py index 186f87b87..8226ef719 100644 --- a/src/everos/infra/persistence/lancedb/repos/__init__.py +++ b/src/everos/infra/persistence/lancedb/repos/__init__.py @@ -11,6 +11,7 @@ from everos.infra.persistence.lancedb.repos import ( episode_repo, atomic_fact_repo, + decision_repo, foresight_repo, agent_case_repo, agent_skill_repo, @@ -24,17 +25,21 @@ from .agent_case import agent_case_repo as agent_case_repo from .agent_skill import agent_skill_repo as agent_skill_repo from .atomic_fact import atomic_fact_repo as atomic_fact_repo +from .decision import decision_repo as decision_repo from .episode import episode_repo as episode_repo from .foresight import foresight_repo as foresight_repo from .knowledge_topic import knowledge_topic_repo as knowledge_topic_repo +from .principle import principle_repo as principle_repo from .user_profile import user_profile_repo as user_profile_repo __all__ = [ "agent_case_repo", "agent_skill_repo", "atomic_fact_repo", + "decision_repo", "episode_repo", "foresight_repo", "knowledge_topic_repo", + "principle_repo", "user_profile_repo", ] diff --git a/src/everos/infra/persistence/lancedb/repos/decision.py b/src/everos/infra/persistence/lancedb/repos/decision.py new file mode 100644 index 000000000..4c6f5ff6a --- /dev/null +++ b/src/everos/infra/persistence/lancedb/repos/decision.py @@ -0,0 +1,20 @@ +"""LanceDB repo singleton for the ``decision`` table.""" + +from __future__ import annotations + +from lancedb import AsyncTable + +from everos.core.persistence.lancedb import LanceDailyLogRepoBase + +from ..lancedb_manager import get_table +from ..tables.decision import Decision + + +class _DecisionRepo(LanceDailyLogRepoBase[Decision]): + schema = Decision + + async def _table_lookup(self) -> AsyncTable: + return await get_table(self.schema.TABLE_NAME, self.schema) + + +decision_repo = _DecisionRepo() diff --git a/src/everos/infra/persistence/lancedb/repos/principle.py b/src/everos/infra/persistence/lancedb/repos/principle.py new file mode 100644 index 000000000..4bff30880 --- /dev/null +++ b/src/everos/infra/persistence/lancedb/repos/principle.py @@ -0,0 +1,20 @@ +"""LanceDB repo singleton for the ``principle`` table.""" + +from __future__ import annotations + +from lancedb import AsyncTable + +from everos.core.persistence.lancedb import LanceRepoBase + +from ..lancedb_manager import get_table +from ..tables.principle import Principle + + +class _PrincipleRepo(LanceRepoBase[Principle]): + schema = Principle + + async def _table_lookup(self) -> AsyncTable: + return await get_table(self.schema.TABLE_NAME, self.schema) + + +principle_repo = _PrincipleRepo() diff --git a/src/everos/infra/persistence/lancedb/tables/__init__.py b/src/everos/infra/persistence/lancedb/tables/__init__.py index 79bdb43e7..f95dabfde 100644 --- a/src/everos/infra/persistence/lancedb/tables/__init__.py +++ b/src/everos/infra/persistence/lancedb/tables/__init__.py @@ -8,10 +8,12 @@ from everos.infra.persistence.lancedb.tables import ( Episode, AtomicFact, + Decision, Foresight, AgentCase, AgentSkill, UserProfile, + Principle, KnowledgeTopic, ParentType, ) @@ -21,18 +23,22 @@ from .agent_case import AgentCase as AgentCase from .agent_skill import AgentSkill as AgentSkill from .atomic_fact import AtomicFact as AtomicFact +from .decision import Decision as Decision from .episode import Episode as Episode from .foresight import Foresight as Foresight from .knowledge_topic import KnowledgeTopic as KnowledgeTopic +from .principle import Principle as Principle from .user_profile import UserProfile as UserProfile __all__ = [ "AgentCase", "AgentSkill", "AtomicFact", + "Decision", "Episode", "Foresight", "KnowledgeTopic", "ParentType", + "Principle", "UserProfile", ] diff --git a/src/everos/infra/persistence/lancedb/tables/decision.py b/src/everos/infra/persistence/lancedb/tables/decision.py new file mode 100644 index 000000000..7df985ee1 --- /dev/null +++ b/src/everos/infra/persistence/lancedb/tables/decision.py @@ -0,0 +1,77 @@ +"""LanceDB ``decision`` table schema. + +Field set for the decision LanceDB row. Each row records one committed +trade-off extracted from a MemCell (title, decision body, reason, +optional impact, tags). ``deprecated_by`` is present from day one so +Decision reflection can soft-deprecate a superseded entry. +""" + +from __future__ import annotations + +import datetime as _dt +from typing import ClassVar + +from everos.core.persistence.lancedb import BaseLanceTable, Vector + +from ._parent_type import ParentType + +_DIM = 1024 + + +class Decision(BaseLanceTable): + """One decision record indexed in LanceDB.""" + + TABLE_NAME: ClassVar[str] = "decision" + BM25_FIELDS: ClassVar[list[str]] = ["decision_tokens", "reason_tokens"] + + id: str + """PK = ``_``.""" + + entry_id: str + """md-side seq id ``dc__``.""" + + owner_id: str + owner_type: str + app_id: str = "default" + project_id: str = "default" + """App / project scope (default ``"default"``); cascade fills from md path.""" + session_id: str | None = None + timestamp: _dt.datetime + + parent_type: str = ParentType.MEMCELL.value + """Source pointer — always :attr:`ParentType.MEMCELL` for decision.""" + + parent_id: str + """Source memcell id.""" + + title: str + decision: str + """Decision body — original surface form (returned for display). + Cascade embed and backfill Phase 1 both read this column.""" + + reason: str + """Why the trade-off was made — original surface form.""" + + impact: str | None = None + tags: list[str] + """Caller-supplied labels (not conversation ``sender_ids``).""" + + decision_tokens: str + """App-layer pre-tokenised ``decision`` text — space-joined tokens. + Primary BM25 column (whitespace tokenizer).""" + + reason_tokens: str + """App-layer pre-tokenised ``reason`` — secondary BM25 column. + Required whenever ``reason`` is (domain ``reason`` is required).""" + + md_path: str + content_sha256: str + """SHA-256 hex digest over the **content-bearing fields only** of + the md entry. Audit inline (owner_id / session_id / timestamp / + parent_id) is NOT in the hash. The exact key set is owned by the + cascade handler's ``content_change_keys`` (lands with the handler).""" + + vector: Vector(_DIM) | None = None # type: ignore[valid-type] + deprecated_by: str | None = None + """Soft-delete marker set by Decision reflection. Value is the + superseding entry_id. ``NULL`` means the row is still active.""" diff --git a/src/everos/infra/persistence/lancedb/tables/principle.py b/src/everos/infra/persistence/lancedb/tables/principle.py new file mode 100644 index 000000000..41435763c --- /dev/null +++ b/src/everos/infra/persistence/lancedb/tables/principle.py @@ -0,0 +1,54 @@ +"""LanceDB ``principle`` table schema. + +Principle is Meta Memory: one ``users//principles.md`` per +user, replaced wholesale on edit (same storage strategy as +``user.md``). Cascade explodes the frontmatter list into one Lance row +per principle. There is no vector and no BM25 — recall is KV-by-owner +(``include_principles``), not HYBRID. + +``source_entry_ids`` is ``list[str]`` (Decision daily-log entry ids), +which Lance can store. The frontmatter's ``list[dict]`` of principles +never lands as a single column. +""" + +from __future__ import annotations + +from typing import ClassVar + +from everos.core.persistence.lancedb import BaseLanceTable + + +class Principle(BaseLanceTable): + """One engineering principle indexed in LanceDB.""" + + TABLE_NAME: ClassVar[str] = "principle" + # No BM25: principle recall is KV-by-owner, not keyword search. + + id: str + """PK = ``_``.""" + + principle_id: str + """md-side id ``pr_<12hex>``, minted at write time.""" + + owner_id: str + owner_type: str + """Always ``"user"`` for this schema.""" + + app_id: str = "default" + project_id: str = "default" + """App / project scope (default ``"default"``); cascade fills from md path.""" + + title: str + statement: str + source_entry_ids: list[str] + """Decision daily-log entry ids this principle was synthesised from.""" + + timestamp_ms: int + """Algo-emitted principle timestamp (ms epoch). Audit only — not + part of ``content_sha256``.""" + + md_path: str + content_sha256: str + """SHA-256 over title + statement + source_entry_ids. Matches → + cascade skips re-upsert of that row. ``timestamp_ms`` is not in + the hash.""" diff --git a/src/everos/infra/persistence/markdown/__init__.py b/src/everos/infra/persistence/markdown/__init__.py index af4f39d85..650d986f0 100644 --- a/src/everos/infra/persistence/markdown/__init__.py +++ b/src/everos/infra/persistence/markdown/__init__.py @@ -17,6 +17,7 @@ BaseDailyWriter, BaseDailyReader, EpisodeWriter, EpisodeReader, EpisodeDailyFrontmatter, AtomicFactDailyFrontmatter, + DecisionDailyFrontmatter, DecisionWriter, DecisionReader, ForesightDailyFrontmatter, AgentCaseDailyFrontmatter, AgentSkillFrontmatter, AgentSkillWriter, AgentSkillReader, @@ -31,15 +32,21 @@ from .mds import AgentCaseDailyFrontmatter as AgentCaseDailyFrontmatter from .mds import AgentSkillFrontmatter as AgentSkillFrontmatter from .mds import AtomicFactDailyFrontmatter as AtomicFactDailyFrontmatter +from .mds import DecisionDailyFrontmatter as DecisionDailyFrontmatter from .mds import EpisodeDailyFrontmatter as EpisodeDailyFrontmatter from .mds import ForesightDailyFrontmatter as ForesightDailyFrontmatter from .mds import KnowledgeDocumentFrontmatter as KnowledgeDocumentFrontmatter from .mds import KnowledgeTopicFrontmatter as KnowledgeTopicFrontmatter +from .mds import PrincipleFrontmatter as PrincipleFrontmatter +from .mds import PrincipleItem as PrincipleItem from .mds import UserProfileFrontmatter as UserProfileFrontmatter +from .mds import mint_principle_id as mint_principle_id +from .mds import render_principles_body as render_principles_body from .readers import AgentCaseReader as AgentCaseReader from .readers import AgentSkillReader as AgentSkillReader from .readers import AtomicFactReader as AtomicFactReader from .readers import BaseDailyReader as BaseDailyReader +from .readers import DecisionReader as DecisionReader from .readers import EpisodeReader as EpisodeReader from .readers import ForesightReader as ForesightReader from .readers import ProfileReader as ProfileReader @@ -49,6 +56,7 @@ from .writers import AgentSkillWriter as AgentSkillWriter from .writers import AtomicFactWriter as AtomicFactWriter from .writers import BaseDailyWriter as BaseDailyWriter +from .writers import DecisionWriter as DecisionWriter from .writers import EpisodeWriter as EpisodeWriter from .writers import ForesightWriter as ForesightWriter from .writers import KnowledgeWriter as KnowledgeWriter @@ -66,6 +74,9 @@ "AtomicFactWriter", "BaseDailyReader", "BaseDailyWriter", + "DecisionDailyFrontmatter", + "DecisionReader", + "DecisionWriter", "EpisodeDailyFrontmatter", "EpisodeReader", "EpisodeWriter", @@ -75,9 +86,13 @@ "KnowledgeDocumentFrontmatter", "KnowledgeTopicFrontmatter", "KnowledgeWriter", + "PrincipleFrontmatter", + "PrincipleItem", "ProfileReader", "ProfileWriter", "UserProfileFrontmatter", "ensure_taxonomy", + "mint_principle_id", "parse_taxonomy", + "render_principles_body", ] diff --git a/src/everos/infra/persistence/markdown/mds/__init__.py b/src/everos/infra/persistence/markdown/mds/__init__.py index 978e3db06..e858bec7e 100644 --- a/src/everos/infra/persistence/markdown/mds/__init__.py +++ b/src/everos/infra/persistence/markdown/mds/__init__.py @@ -26,21 +26,31 @@ from .agent_case import AgentCaseDailyFrontmatter as AgentCaseDailyFrontmatter from .agent_skill import AgentSkillFrontmatter as AgentSkillFrontmatter from .atomic_fact import AtomicFactDailyFrontmatter as AtomicFactDailyFrontmatter +from .decision import DecisionDailyFrontmatter as DecisionDailyFrontmatter from .episode import EpisodeDailyFrontmatter as EpisodeDailyFrontmatter from .foresight import ForesightDailyFrontmatter as ForesightDailyFrontmatter from .knowledge_document import ( KnowledgeDocumentFrontmatter as KnowledgeDocumentFrontmatter, ) from .knowledge_topic import KnowledgeTopicFrontmatter as KnowledgeTopicFrontmatter +from .principle import PrincipleFrontmatter as PrincipleFrontmatter +from .principle import PrincipleItem as PrincipleItem +from .principle import mint_principle_id as mint_principle_id +from .principle import render_principles_body as render_principles_body from .profile import UserProfileFrontmatter as UserProfileFrontmatter __all__ = [ "AgentCaseDailyFrontmatter", "AgentSkillFrontmatter", "AtomicFactDailyFrontmatter", + "DecisionDailyFrontmatter", "EpisodeDailyFrontmatter", "ForesightDailyFrontmatter", "KnowledgeDocumentFrontmatter", "KnowledgeTopicFrontmatter", + "PrincipleFrontmatter", + "PrincipleItem", "UserProfileFrontmatter", + "mint_principle_id", + "render_principles_body", ] diff --git a/src/everos/infra/persistence/markdown/mds/decision.py b/src/everos/infra/persistence/markdown/mds/decision.py new file mode 100644 index 000000000..f64f0245e --- /dev/null +++ b/src/everos/infra/persistence/markdown/mds/decision.py @@ -0,0 +1,36 @@ +"""Decision frontmatter — daily-log markdown for user-scoped decisions. + +Path: ``users//decisions/decision-.md``. + +User-readable (same convention as episodes), not a dot-prefixed internal +directory. ``deprecated_entries`` is present from day one so Decision +reflection can soft-deprecate a superseded entry without rewriting history. +""" + +from __future__ import annotations + +import datetime as _dt +from typing import ClassVar, Literal + +from pydantic import Field + +from everos.core.persistence.markdown import ( + DailyLogPathMixin, + UserScopedFrontmatter, +) + + +class DecisionDailyFrontmatter(DailyLogPathMixin, UserScopedFrontmatter): + """Frontmatter for ``users//decisions/decision-.md``.""" + + ENTRY_ID_PREFIX: ClassVar[str] = "dc" + DIR_NAME: ClassVar[str] = "decisions" + FILE_PREFIX: ClassVar[str] = "decision" + + type: Literal["decision_daily"] = "decision_daily" + file_type: Literal["decision_daily"] = "decision_daily" + date: _dt.date + entry_count: int = 0 + created_at: _dt.datetime | None = None + last_appended_at: _dt.datetime | None = None + deprecated_entries: dict[str, str] = Field(default_factory=dict) diff --git a/src/everos/infra/persistence/markdown/mds/principle.py b/src/everos/infra/persistence/markdown/mds/principle.py new file mode 100644 index 000000000..7bb531d0a --- /dev/null +++ b/src/everos/infra/persistence/markdown/mds/principle.py @@ -0,0 +1,70 @@ +"""Principle frontmatter — single-file engineering principles per user. + +Path: ``users//principles.md``. + +Principle is Meta Memory synthesised from a Decision cluster, not a +product Memory Kind. Storage reuses the profile chassis (fixed-name +single-file rewrite via :class:`ProfileWriter` / :class:`ProfileReader`); +this schema only supplies ``PROFILE_FILENAME`` plus the structured +``principles`` list. + +LanceDB has no ``list[dict]`` column, so the list stays in frontmatter +and the cascade handler explodes it into one ``principle`` row per item. +The markdown body is a human-readable list and is not indexed. +""" + +from __future__ import annotations + +import uuid +from typing import ClassVar, Literal + +from pydantic import BaseModel, Field + +from everos.core.persistence.markdown import ProfilePathMixin, UserScopedFrontmatter + + +def mint_principle_id() -> str: + """Mint a fresh principle id (``pr_<12hex>``). + + EverAlgo ``Principle`` has no id; EverOS assigns one at write time + so cascade can explode a stable Lance PK ``_``. + """ + return f"pr_{uuid.uuid4().hex[:12]}" + + +class PrincipleItem(BaseModel): + """One engineering principle in the ``principles.md`` frontmatter list. + + ``id`` is the EverOS-minted ``pr_<12hex>`` (EverAlgo ``Principle`` has + no id). ``source_entry_ids`` point at Decision daily-log entry ids. + """ + + id: str + title: str + statement: str + source_entry_ids: list[str] = Field(default_factory=list) + timestamp_ms: int = 0 + + +class PrincipleFrontmatter(ProfilePathMixin, UserScopedFrontmatter): + """Frontmatter for ``users//principles.md``.""" + + PROFILE_FILENAME: ClassVar[str] = "principles.md" + + type: Literal["principle"] = "principle" + + principles: list[PrincipleItem] = Field(default_factory=list) + """Structured principle list. Cascade explodes each item into one + Lance ``principle`` row (``id = _``).""" + + +def render_principles_body(items: list[PrincipleItem]) -> str: + """Human-readable markdown list for the ``principles.md`` body. + + Display-only: cascade indexes the structured frontmatter list, not + this body. Empty input yields the empty string (no trailing newline). + """ + if not items: + return "" + lines = [f"- **{item.title}.** {item.statement}" for item in items] + return "\n".join(lines) + "\n" diff --git a/src/everos/infra/persistence/markdown/readers/__init__.py b/src/everos/infra/persistence/markdown/readers/__init__.py index 342bb5080..ac4b5bca1 100644 --- a/src/everos/infra/persistence/markdown/readers/__init__.py +++ b/src/everos/infra/persistence/markdown/readers/__init__.py @@ -34,6 +34,7 @@ from .agent_skill_reader import AgentSkillReader as AgentSkillReader from .atomic_fact_reader import AtomicFactReader as AtomicFactReader from .base import BaseDailyReader as BaseDailyReader +from .decision_reader import DecisionReader as DecisionReader from .episode_reader import EpisodeReader as EpisodeReader from .foresight_reader import ForesightReader as ForesightReader from .profile_reader import ProfileReader as ProfileReader @@ -45,6 +46,7 @@ "AgentSkillReader", "AtomicFactReader", "BaseDailyReader", + "DecisionReader", "EpisodeReader", "ForesightReader", "ProfileReader", diff --git a/src/everos/infra/persistence/markdown/readers/decision_reader.py b/src/everos/infra/persistence/markdown/readers/decision_reader.py new file mode 100644 index 000000000..8bafc6c50 --- /dev/null +++ b/src/everos/infra/persistence/markdown/readers/decision_reader.py @@ -0,0 +1,31 @@ +"""Decision daily-log reader — symmetric with :class:`DecisionWriter`.""" + +from __future__ import annotations + +import datetime as _dt +from pathlib import Path + +from everos.core.persistence import MemoryRoot + +from ..mds import DecisionDailyFrontmatter +from .base import BaseDailyReader + + +class DecisionReader(BaseDailyReader): + """Read decision daily-log files.""" + + schema = DecisionDailyFrontmatter + + def __init__(self, root: MemoryRoot) -> None: + super().__init__(root) + + def path_for( + self, + owner_id: str, + date: _dt.date | None = None, + *, + app_id: str = "default", + project_id: str = "default", + ) -> Path: + """Resolve the decision daily-log path under the / prefix.""" + return super().path_for(owner_id, date, app_id=app_id, project_id=project_id) diff --git a/src/everos/infra/persistence/markdown/writers/__init__.py b/src/everos/infra/persistence/markdown/writers/__init__.py index 377f7f630..2d8b58d43 100644 --- a/src/everos/infra/persistence/markdown/writers/__init__.py +++ b/src/everos/infra/persistence/markdown/writers/__init__.py @@ -4,7 +4,8 @@ here: * :class:`BaseDailyWriter` — daily-log append (episode / atomic - fact / foresight / agent case). Subclass and bind ``schema``. + fact / decision / foresight / agent case). Subclass and bind + ``schema``. * :class:`AgentSkillWriter` — directory + progressive disclosure (``skills/skill_/{SKILL.md, references/, scripts/}``). Single class, no subclassing. @@ -28,6 +29,7 @@ from .agent_skill_writer import AgentSkillWriter as AgentSkillWriter from .atomic_fact_writer import AtomicFactWriter as AtomicFactWriter from .base import BaseDailyWriter as BaseDailyWriter +from .decision_writer import DecisionWriter as DecisionWriter from .episode_writer import EpisodeWriter as EpisodeWriter from .foresight_writer import ForesightWriter as ForesightWriter from .knowledge_writer import KnowledgeWriter as KnowledgeWriter @@ -38,6 +40,7 @@ "AgentSkillWriter", "AtomicFactWriter", "BaseDailyWriter", + "DecisionWriter", "EpisodeWriter", "ForesightWriter", "KnowledgeWriter", diff --git a/src/everos/infra/persistence/markdown/writers/decision_writer.py b/src/everos/infra/persistence/markdown/writers/decision_writer.py new file mode 100644 index 000000000..16e5d266e --- /dev/null +++ b/src/everos/infra/persistence/markdown/writers/decision_writer.py @@ -0,0 +1,62 @@ +"""Decision daily-log writer — md is the SoT for decisions. + +Caller hands pre-built ``inline`` (``owner_id`` / ``session_id`` / +``timestamp`` / ``parent_type`` / ``parent_id`` / ``tags``) plus the +``Title`` / ``Decision`` / ``Reason`` / optional ``Impact`` sections. +The chassis manages the in-file ``entry_id`` sequence +(``dc__``). ``append_entry`` / ``append_entries`` come +from :class:`BaseDailyWriter`; this subclass only declares the schema +and the per-schema frontmatter / counter hooks. + +Domain → ``(inline, sections)`` shaping lives in the calling strategy +(infra must not import ``memory``). +""" + +from __future__ import annotations + +import datetime as _dt +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +import anyio + +from everos.component.utils.datetime import ( + get_now_with_timezone, + to_iso_format, +) +from everos.core.persistence import MarkdownReader + +from ..mds import DecisionDailyFrontmatter +from .base import BaseDailyWriter + + +class DecisionWriter(BaseDailyWriter): + """Daily-log writer for the Decision schema (md = SoT).""" + + schema = DecisionDailyFrontmatter + + def _frontmatter_updates( + self, + scope_id: str, + date: _dt.date, + *, + next_count: int, + ) -> Mapping[str, Any] | None: + return { + "id": f"decision_log_{scope_id}_{date.isoformat()}", + "type": "decision_daily", + "file_type": "decision_daily", + "schema_version": 1, + "user_id": scope_id, + "track": "user", + "date": date.isoformat(), + "entry_count": next_count, + "last_appended_at": to_iso_format(get_now_with_timezone()), + } + + async def _current_count(self, path: Path) -> int: + if not await anyio.Path(path).is_file(): + return 0 + parsed = await MarkdownReader.read(path) + return parsed.frontmatter.get("entry_count", 0) diff --git a/src/everos/infra/persistence/markdown/writers/profile_writer.py b/src/everos/infra/persistence/markdown/writers/profile_writer.py index 76e8d00c7..948fe64e2 100644 --- a/src/everos/infra/persistence/markdown/writers/profile_writer.py +++ b/src/everos/infra/persistence/markdown/writers/profile_writer.py @@ -5,6 +5,7 @@ filename under the agent or user directory:: users//user.md ← user profile + users//principles.md ← engineering principles (meta) users//behaviors.md ← user behaviour patterns agents//agent.md ← agent playbook agents//soul.md ← agent identity / values diff --git a/src/everos/memory/__init__.py b/src/everos/memory/__init__.py index a5ded6fe6..90491ed90 100644 --- a/src/everos/memory/__init__.py +++ b/src/everos/memory/__init__.py @@ -20,11 +20,13 @@ from .models import AgentCase as AgentCase from .models import AlgoAgentCase as AlgoAgentCase from .models import AlgoAtomicFact as AlgoAtomicFact +from .models import AlgoDecision as AlgoDecision from .models import AlgoEpisode as AlgoEpisode from .models import AlgoForesight as AlgoForesight from .models import AlgoMessage as AlgoMessage from .models import AtomicFact as AtomicFact from .models import CanonicalMessage as CanonicalMessage +from .models import Decision as Decision from .models import Episode as Episode from .models import Foresight as Foresight from .models import IngestResult as IngestResult @@ -36,11 +38,13 @@ "AgentCase", "AlgoAgentCase", "AlgoAtomicFact", + "AlgoDecision", "AlgoEpisode", "AlgoForesight", "AlgoMessage", "AtomicFact", "CanonicalMessage", + "Decision", "Episode", "Foresight", "IngestResult", diff --git a/src/everos/memory/cascade/_backfill.py b/src/everos/memory/cascade/_backfill.py index 767ee6d24..121b18e33 100644 --- a/src/everos/memory/cascade/_backfill.py +++ b/src/everos/memory/cascade/_backfill.py @@ -8,7 +8,7 @@ - Phase 1 (:func:`_run_phase_vectors`) — re-embed missing vectors on existing rows. - Phase 2 (:func:`_run_phase_clusters`) — build clusters on the newly- - embedded episodes / agent cases. + embedded episodes / agent cases / decisions. - Phase 3 (:func:`_run_phase_skills`) — extract agent skills from clustered cases. @@ -51,12 +51,14 @@ AgentCase, AgentSkill, AtomicFact, + Decision, Episode, Foresight, KnowledgeTopic, agent_case_repo, agent_skill_repo, atomic_fact_repo, + decision_repo, episode_repo, foresight_repo, get_table, @@ -69,11 +71,13 @@ ) from everos.memory.events import ( AgentCaseExtracted, + DecisionExtracted, EpisodeExtracted, SkillClusterUpdated, ) from everos.memory.strategies import ( extract_agent_skill, + trigger_decision_clustering, trigger_profile_clustering, trigger_skill_clustering, ) @@ -147,7 +151,9 @@ def capability_missing( ) -> None: ... def server_running(self) -> None: ... def estimate_vectors(self, rows: int, tokens: int) -> None: ... - def estimate_clusters(self, episodes: int, cases: int) -> None: ... + def estimate_clusters( + self, episodes: int, cases: int, *, decisions: int = 0 + ) -> None: ... def estimate_skills(self, cases: int, clusters: int) -> None: ... async def confirm(self, prompt: str, *, auto_yes: bool) -> bool: ... def emit_progress(self, done: int, total: int) -> None: ... @@ -182,7 +188,9 @@ def server_running(self) -> None: def estimate_vectors(self, rows: int, tokens: int) -> None: return None - def estimate_clusters(self, episodes: int, cases: int) -> None: + def estimate_clusters( + self, episodes: int, cases: int, *, decisions: int = 0 + ) -> None: return None def estimate_skills(self, cases: int, clusters: int) -> None: @@ -242,6 +250,7 @@ def _agent_skill_embed_text(row: dict[str, Any]) -> str: subject_of=lambda r: r.get("subject") or None, ), _TableSpec(AtomicFact, atomic_fact_repo, lambda r: r["fact"]), + _TableSpec(Decision, decision_repo, lambda r: r["decision"]), _TableSpec(Foresight, foresight_repo, lambda r: r["foresight"]), _TableSpec(AgentCase, agent_case_repo, lambda r: r["task_intent"]), _TableSpec(AgentSkill, agent_skill_repo, _agent_skill_embed_text), @@ -899,7 +908,8 @@ class _ClusterPhaseResult: """Phase 2 outcome. ``events_emitted`` counts every synthesized ``EpisodeExtracted`` + - ``AgentCaseExtracted`` event fanned into the ephemeral engine. + ``AgentCaseExtracted`` + ``DecisionExtracted`` event fanned into the + ephemeral engine. ``clusters_before`` / ``clusters_after`` are the total ``cluster`` row count (:meth:`_ClusterRepo.count`, across every owner/kind) taken immediately before dispatch and after the engine drains, so @@ -922,8 +932,9 @@ async def _scan_all_rows(schema: type[BaseLanceTable]) -> list[dict[str, Any]]: Phase 2 doesn't care whether a row already carries a vector — the cluster strategies re-embed the row's text themselves (see - ``trigger_profile_clustering`` / ``trigger_skill_clustering``); it - just needs every existing episode / agent case so it can + ``trigger_profile_clustering`` / ``trigger_skill_clustering`` / + ``trigger_decision_clustering``); it + just needs every existing episode / agent case / decision so it can synthesize the trigger event Tier 1's gated-off strategies never emitted (embed-requiring strategies are body-guarded off when ``get_embedding_capability().available`` is false — see @@ -981,6 +992,34 @@ def _agent_case_row_to_event(raw: dict[str, Any]) -> AgentCaseExtracted: ) +def _decision_row_to_event(raw: dict[str, Any]) -> DecisionExtracted: + """Synthesize the ``DecisionExtracted(source="pipeline")`` this row's + original pipeline run never emitted — Tier 1 gated + ``trigger_decision_clustering`` off before it could fire. + + ``event_id`` carries a ``backfill_`` prefix so ops can tell a + synthesized run apart from a real one in logs / run records. + """ + tags_raw = raw.get("tags") or [] + tags = [str(t) for t in tags_raw] + return DecisionExtracted( + event_id=f"backfill_{uuid4().hex}", + memcell_id=raw["parent_id"], + decision_entry_id=raw["entry_id"], + title=raw.get("title") or "", + decision_text=raw["decision"], + reason=raw.get("reason") or "", + impact=raw.get("impact"), + tags=tags, + decision_timestamp_ms=to_timestamp_ms(raw["timestamp"]), + owner_id=raw["owner_id"], + session_id=raw.get("session_id"), + app_id=raw.get("app_id", "default"), + project_id=raw.get("project_id", "default"), + source="pipeline", + ) + + def _report_emit_progress(presenter: BackfillPresenter, done: int, total: int) -> None: if done % _PROGRESS_EVERY == 0 or done == total: presenter.emit_progress(done, total) @@ -992,6 +1031,7 @@ async def _emit_synthetic_events( cases: list[dict[str, Any]], *, presenter: BackfillPresenter, + decisions: list[dict[str, Any]] | None = None, ) -> int: """Fan every row into ``engine`` as its own synthetic trigger event. @@ -1000,20 +1040,23 @@ async def _emit_synthetic_events( on a partially-completed root) would re-cluster the same rows and grow cluster counts spuriously. ``cluster_repo.find_cluster_id_for_member`` is O(log N) via the reverse index and is the exact primitive for - this dedup. ``member_type`` values (``"episode"`` / ``"case"``) match - what :func:`trigger_profile_clustering` and - :func:`trigger_skill_clustering` insert on the write path — a mismatch - here would silently disable the skip and re-open the double-cluster - window. - - Episodes first, then agent cases — order doesn't affect correctness - (each event routes to its own strategy independently); it only - keeps the progress readout monotonic. The progress counter advances - for skipped rows too so the readout matches the pre-scan estimate; - ``_ClusterPhaseResult.events_emitted`` reflects "rows processed" - (real emits + already-clustered skips), not "engine.emit calls". + this dedup. ``member_type`` values (``"episode"`` / ``"case"`` / + ``"decision"``) match what :func:`trigger_profile_clustering`, + :func:`trigger_skill_clustering`, and + :func:`trigger_decision_clustering` insert on the write path — a + mismatch here would silently disable the skip and re-open the + double-cluster window. + + Episodes first, then agent cases, then decisions — order doesn't + affect correctness (each event routes to its own strategy + independently); it only keeps the progress readout monotonic. The + progress counter advances for skipped rows too so the readout + matches the pre-scan estimate; ``_ClusterPhaseResult.events_emitted`` + reflects "rows processed" (real emits + already-clustered skips), + not "engine.emit calls". """ - total = len(episodes) + len(cases) + decision_rows = decisions or [] + total = len(episodes) + len(cases) + len(decision_rows) emitted = 0 for raw in episodes: # entry_id is only per-owner unique — scope the reverse lookup @@ -1042,6 +1085,18 @@ async def _emit_synthetic_events( await engine.emit(_agent_case_row_to_event(raw)) emitted += 1 _report_emit_progress(presenter, emitted, total) + for raw in decision_rows: + existing = await cluster_repo.find_cluster_id_for_member( + member_type="decision", + member_id=raw["entry_id"], + app_id=raw["app_id"], + project_id=raw["project_id"], + owner_id=raw["owner_id"], + ) + if existing is None: + await engine.emit(_decision_row_to_event(raw)) + emitted += 1 + _report_emit_progress(presenter, emitted, total) return emitted @@ -1100,7 +1155,7 @@ def _build_cluster_engine() -> OfflineEngine: ``memory`` may not import ``service`` (the layering rule forbids it — see ``.claude/rules/architecture.md``), so this cannot reuse ``service.memorize``'s process-wide engine singleton; it builds its - own instance and registers only the two clustering strategies whose + own instance and registers only the three clustering strategies whose body-guards short-circuit under Tier 1 (no embedding provider). It shares the live engine's ``ome_db`` jobstore path, so if a server is already running against the same @@ -1131,6 +1186,7 @@ def _build_cluster_engine() -> OfflineEngine: ) engine.register(trigger_profile_clustering) engine.register(trigger_skill_clustering) + engine.register(trigger_decision_clustering) return engine @@ -1155,16 +1211,17 @@ async def _ensure_cluster_schema() -> None: async def _run_phase_clusters( *, auto_yes: bool, presenter: BackfillPresenter ) -> _ClusterPhaseResult: - """Rebuild clusters for every episode / agent case via synthetic events. - - Scan every episode + agent case row → surface the estimate → - confirm once → synthesize the ``EpisodeExtracted`` / - ``AgentCaseExtracted`` event each row's original pipeline run - would have emitted had the clustering strategies not been gated - off under Tier 1 (their embed-requiring body-guards short-circuited - the dispatch) → replay them through a dedicated - engine that registers only those two (now-eligible, since embed - is available) strategies → wait for the engine to drain. + """Rebuild clusters for every episode / agent case / decision via + synthetic events. + + Scan every episode + agent case + decision row → surface the + estimate → confirm once → synthesize the ``EpisodeExtracted`` / + ``AgentCaseExtracted`` / ``DecisionExtracted`` event each row's + original pipeline run would have emitted had the clustering + strategies not been gated off under Tier 1 (their embed-requiring + body-guards short-circuited the dispatch) → replay them through a + dedicated engine that registers only those three (now-eligible, + since embed is available) strategies → wait for the engine to drain. Idempotent: :func:`_emit_synthetic_events` filters each row through :meth:`cluster_repo.find_cluster_id_for_member` before emitting, so @@ -1172,14 +1229,13 @@ async def _run_phase_clusters( Ctrl-C interruption) skips rows already attached to a cluster. Cluster counts stop growing spuriously across reruns. - Episode rows carrying ``parent_type == "cluster"`` are Reflection's - merged episodes (``orchestrator._write_merged_episode``), not source - pipeline events, and are excluded — mirrors the same - ``parent_type == "memcell"`` filter idiom used by - ``extract_user_profile._select_via_cluster``. Synthesizing an event - for one would carry a bogus ``memcell_id`` (actually a cluster id) - and defeat ``trigger_profile_clustering``'s own - ``applies_to=lambda e: e.source == "pipeline"`` exclusion of + Episode (and later, decision) rows carrying ``parent_type == + "cluster"`` are Reflection merged rows, not source pipeline events, + and are excluded — mirrors the same ``parent_type == "memcell"`` + filter idiom used by ``extract_user_profile._select_via_cluster``. + Synthesizing an event for one would carry a bogus ``memcell_id`` + (actually a cluster id) and defeat ``trigger_profile_clustering`` / + ``trigger_decision_clustering`` ``applies_to`` exclusion of Reflection output. """ # Preflight capability + OME lock BEFORE any collection work. Both @@ -1206,17 +1262,23 @@ async def _run_phase_clusters( if row.get("parent_type") == "memcell" ] cases = await _scan_all_rows(AgentCase) - total = len(episodes) + len(cases) + decisions = [ + row + for row in await _scan_all_rows(Decision) + if row.get("parent_type") == "memcell" + ] + total = len(episodes) + len(cases) + len(decisions) if total == 0: presenter.nothing_to_backfill( - "Nothing to backfill. No episodes or agent cases found." + "Nothing to backfill. No episodes, agent cases, or decisions found." ) return _ClusterPhaseResult() - presenter.estimate_clusters(len(episodes), len(cases)) + presenter.estimate_clusters(len(episodes), len(cases), decisions=len(decisions)) if not await presenter.confirm( f"proceed with {total:,} memories " - f"({len(episodes):,} episodes, {len(cases):,} agent cases)", + f"({len(episodes):,} episodes, {len(cases):,} agent cases, " + f"{len(decisions):,} decisions)", auto_yes=auto_yes, ): return _ClusterPhaseResult(aborted=True) @@ -1235,7 +1297,11 @@ async def _run_phase_clusters( return _ClusterPhaseResult(aborted=True, blocked_by_server=True) try: emitted = await _emit_synthetic_events( - engine, episodes, cases, presenter=presenter + engine, + episodes, + cases, + presenter=presenter, + decisions=decisions, ) drained = await engine.wait_idle(timeout=_CLUSTER_WAIT_TIMEOUT_SECONDS) if not drained: diff --git a/src/everos/memory/cascade/handlers/__init__.py b/src/everos/memory/cascade/handlers/__init__.py index b9306d74c..5b4d9b683 100644 --- a/src/everos/memory/cascade/handlers/__init__.py +++ b/src/everos/memory/cascade/handlers/__init__.py @@ -1,12 +1,13 @@ """Cascade handlers — one per kind, sharing the :class:`Handler` chassis. -Four daily-log handlers (episode / atomic_fact / foresight / +Five daily-log handlers (episode / atomic_fact / foresight / decision / agent_case) inherit :class:`BaseDailyLogHandler` for the shared read / diff / upsert / delete loop; the per-kind subclass only declares its repo binding and ``_build_row`` mapping. ``agent_skill``, -``user_profile``, and ``knowledge_topic`` stand alone — they're -single-file kinds (no entries, no per-entry diff), so they implement -:class:`Handler` directly and own their reconcile loop. +``user_profile``, ``principle``, and ``knowledge_topic`` stand alone — +they're single-file kinds (no entries, no per-entry diff), so they +implement :class:`Handler` directly and own their reconcile loop. +``principle`` explodes one file into N Lance rows. """ from .agent_case import AgentCaseHandler as AgentCaseHandler @@ -14,21 +15,25 @@ from .atomic_fact import AtomicFactHandler as AtomicFactHandler from .base import Handler as Handler from .base import HandlerDeps as HandlerDeps +from .decision import DecisionHandler as DecisionHandler from .episode import EpisodeHandler as EpisodeHandler from .foresight import ForesightHandler as ForesightHandler from .knowledge_document import KnowledgeDocumentHandler as KnowledgeDocumentHandler from .knowledge_topic import KnowledgeTopicHandler as KnowledgeTopicHandler +from .principle import PrincipleHandler as PrincipleHandler from .user_profile import UserProfileHandler as UserProfileHandler __all__ = [ "AgentCaseHandler", "AgentSkillHandler", "AtomicFactHandler", + "DecisionHandler", "EpisodeHandler", "ForesightHandler", "Handler", "HandlerDeps", "KnowledgeDocumentHandler", "KnowledgeTopicHandler", + "PrincipleHandler", "UserProfileHandler", ] diff --git a/src/everos/memory/cascade/handlers/_daily_log_base.py b/src/everos/memory/cascade/handlers/_daily_log_base.py index 8791a418f..0467d8511 100644 --- a/src/everos/memory/cascade/handlers/_daily_log_base.py +++ b/src/everos/memory/cascade/handlers/_daily_log_base.py @@ -1,7 +1,7 @@ """Shared diff / dispatch loop for every daily-log cascade handler. -The 4 daily-log kinds (episode / atomic_fact / foresight / agent_case) -all do the same three-way reconcile against LanceDB: +The 5 daily-log kinds (episode / atomic_fact / foresight / decision / +agent_case) all do the same three-way reconcile against LanceDB: 1. Parse the md into structured entries. 2. Fetch existing rows for the same ``md_path``. @@ -52,7 +52,7 @@ class ParsedEntry: class BaseDailyLogHandler(Handler): - """Common chassis for the 4 daily-log cascade handlers. + """Common chassis for the 5 daily-log cascade handlers. Subclass requirements: diff --git a/src/everos/memory/cascade/handlers/decision.py b/src/everos/memory/cascade/handlers/decision.py new file mode 100644 index 000000000..ec25e2cb9 --- /dev/null +++ b/src/everos/memory/cascade/handlers/decision.py @@ -0,0 +1,105 @@ +"""Decision cascade handler — md → LanceDB ``decision`` table. + +Two-field BM25: ``decision_tokens`` is the primary search column, +``reason_tokens`` rides along from the Reason section. The vector +embedding is fed only from the Decision body (reason / title / impact +are supporting context, not the retrieval anchor). + +md contract (must match ``extract_decision._decision_to_entry_body``): + +``inline`` block: + +- ``owner_id`` / ``session_id`` / ``timestamp`` — same shape as + Episode / Foresight. +- ``parent_id``: source memcell id (``parent_type`` defaults to + ``"memcell"``). +- ``tags`` (optional): list rendering ``[runtime, rust]``. Not + conversation ``sender_ids``. + +``sections``: + +- ``Title``: short label (Lance ``title`` column; hashed so edits + propagate, not embedded). +- ``Decision``: committed trade-off text (embedded + BM25 primary). + Embedding is a soft dependency: when unavailable, ``vector`` is + written as ``None`` and the row stays BM25/scalar-searchable only. +- ``Reason``: why the trade-off was made (secondary BM25 only). +- ``Impact`` (optional): consequence note (display only). +""" + +from __future__ import annotations + +from everos.component.embedding import get_embedding_capability +from everos.core.observability.logging import get_logger +from everos.infra.persistence.lancedb import Decision, ParentType, decision_repo + +from ._common import parse_inline_list, require_iso_timestamp +from ._daily_log_base import BaseDailyLogHandler, ParsedEntry + +logger = get_logger(__name__) + + +class DecisionHandler(BaseDailyLogHandler): + """Cascade handler for ``users//decisions/decision-*.md``.""" + + kind = "decision" + lance_repo = decision_repo + content_change_keys = ( + "section:Title", + "section:Decision", + "section:Reason", + "section:Impact", + "inline:tags", + ) + """Title / Decision / Reason / Impact + tags. Audit inline + (owner_id / session_id / timestamp / parent_id) is excluded — + changes there don't propagate. Title is hashed so the Lance + ``title`` column stays in sync even though it is not embedded.""" + + async def _build_row( + self, + *, + owner_id: str, + owner_type: str, + app_id: str = "default", + project_id: str = "default", + md_path: str, + entry: ParsedEntry, + ) -> Decision: + s = entry.structured + title = s.sections.get("Title", "").strip() + text = s.sections.get("Decision", "").strip() + reason = s.sections.get("Reason", "").strip() + impact = (s.sections.get("Impact") or "").strip() or None + decision_tokens = self._deps.tokenizer.tokenize(text) + reason_tokens = self._deps.tokenizer.tokenize(reason) + vector = await get_embedding_capability().embed_or_none(text) + if vector is None: + logger.debug( + "cascade_handler_embed_skipped", + kind=self.kind, + entry_id=entry.entry_id, + reason="embedding_capability_unavailable", + ) + return Decision( + id=f"{owner_id}_{entry.entry_id}", + entry_id=entry.entry_id, + owner_id=owner_id, + owner_type=owner_type, + app_id=app_id, + project_id=project_id, + session_id=s.inline.get("session_id"), + timestamp=require_iso_timestamp(s.inline.get("timestamp")), + parent_type=s.inline.get("parent_type") or ParentType.MEMCELL.value, + parent_id=s.inline.get("parent_id", ""), + title=title, + decision=text, + reason=reason, + impact=impact, + tags=parse_inline_list(s.inline.get("tags") or ""), + decision_tokens=" ".join(decision_tokens), + reason_tokens=" ".join(reason_tokens), + md_path=md_path, + content_sha256=entry.content_sha256, + vector=vector, + ) diff --git a/src/everos/memory/cascade/handlers/principle.py b/src/everos/memory/cascade/handlers/principle.py new file mode 100644 index 000000000..03491001b --- /dev/null +++ b/src/everos/memory/cascade/handlers/principle.py @@ -0,0 +1,164 @@ +"""Principle cascade handler — md → LanceDB ``principle`` table. + +``principles.md`` is a single-file kind (same chassis as +``users//user.md``): one file per user, replaced wholesale. +Unlike :class:`UserProfileHandler` (one row per file), this handler +explodes the frontmatter ``principles`` list into N Lance rows +(``id = _``). + +md contract: + +- frontmatter: :class:`PrincipleFrontmatter` (``user_id`` + structured + ``principles`` list). +- body: human-readable markdown list (not indexed). + +``timestamp_ms`` is audit: it lands on the row but is excluded from +``content_sha256``, so a timestamp-only drift skips re-upsert. +""" + +from __future__ import annotations + +import json +from typing import Any, ClassVar + +from everos.core.persistence import MarkdownReader +from everos.infra.persistence.lancedb import Principle, principle_repo + +from ..types import HandlerOutcome +from ._common import content_sha256 as compute_content_sha256 +from ._common import resolve_scope +from .base import Handler + + +class PrincipleHandler(Handler): + """Cascade handler for ``users//principles.md``.""" + + kind = "principle" + lance_repo: ClassVar[Any] = principle_repo + """Exposed for ``CascadeWorker._optimize_touched_kinds``.""" + + content_change_keys: ClassVar[tuple[str, ...]] = ( + "frontmatter:title", + "frontmatter:statement", + "frontmatter:source_entry_ids_json", + ) + + async def handle_added_or_modified(self, md_path: str) -> HandlerOutcome: + absolute = self._deps.memory_root.root / md_path + parsed = await MarkdownReader.read(absolute) + fm = parsed.frontmatter + + owner_id = str(fm.get("user_id", "")) + if not owner_id: + raise ValueError( + f"principle md missing required frontmatter user_id: {md_path}" + ) + app_id, project_id = resolve_scope(md_path) + + items = _parse_items(fm.get("principles", [])) + seen_ids: set[str] = set() + desired: dict[str, Principle] = {} + for item in items: + principle_id = str(item.get("id") or "").strip() + if not principle_id: + raise ValueError(f"principle md has an item with empty id: {md_path}") + if principle_id in seen_ids: + raise ValueError( + f"principle md has duplicate id {principle_id!r}: {md_path}" + ) + seen_ids.add(principle_id) + title = str(item.get("title", "")) + statement = str(item.get("statement", "")) + source_entry_ids = _as_str_list(item.get("source_entry_ids", [])) + source_json = json.dumps( + source_entry_ids, sort_keys=True, ensure_ascii=False + ) + digest = compute_content_sha256( + { + "frontmatter:title": title, + "frontmatter:statement": statement, + "frontmatter:source_entry_ids_json": source_json, + } + ) + row_id = f"{owner_id}_{principle_id}" + desired[row_id] = Principle( + id=row_id, + principle_id=principle_id, + owner_id=owner_id, + owner_type="user", + app_id=app_id, + project_id=project_id, + title=title, + statement=statement, + source_entry_ids=source_entry_ids, + timestamp_ms=int(item.get("timestamp_ms") or 0), + md_path=md_path, + content_sha256=digest, + ) + + existing = await principle_repo.find_where( + f"md_path = '{_q(md_path)}'", + limit=10_000, + ) + existing_by_id = {row.id: row for row in existing} + + to_upsert = [ + row + for row_id, row in desired.items() + if existing_by_id.get(row_id) is None + or existing_by_id[row_id].content_sha256 != row.content_sha256 + ] + to_delete_ids = [row.id for row in existing if row.id not in desired] + skipped = len(desired) - len(to_upsert) + + if to_upsert: + await principle_repo.upsert(to_upsert) + if to_delete_ids: + in_list = ", ".join(f"'{_q(rid)}'" for rid in to_delete_ids) + await principle_repo.delete( + f"md_path = '{_q(md_path)}' AND id IN ({in_list})" + ) + + return HandlerOutcome( + md_path=md_path, + kind=self.kind, + upserted=len(to_upsert), + deleted=len(to_delete_ids), + skipped=skipped, + ) + + async def handle_deleted(self, md_path: str) -> HandlerOutcome: + deleted = await principle_repo.delete_by_md_path(md_path) + return HandlerOutcome( + md_path=md_path, + kind=self.kind, + upserted=0, + deleted=deleted, + skipped=0, + ) + + +def _parse_items(raw: Any) -> list[dict[str, Any]]: + if raw is None: + return [] + if not isinstance(raw, list): + raise ValueError("principle frontmatter principles must be a list") + items: list[dict[str, Any]] = [] + for item in raw: + if not isinstance(item, dict): + raise ValueError("principle frontmatter item must be a mapping") + items.append(item) + return items + + +def _as_str_list(raw: Any) -> list[str]: + if raw is None: + return [] + if not isinstance(raw, list): + return [] + return [str(v) for v in raw] + + +def _q(text: str) -> str: + """Defensive SQL-quote escape (mirrors daily-log handler convention).""" + return text.replace("'", "''") diff --git a/src/everos/memory/cascade/registry.py b/src/everos/memory/cascade/registry.py index 8537575e6..06ae3940f 100644 --- a/src/everos/memory/cascade/registry.py +++ b/src/everos/memory/cascade/registry.py @@ -22,26 +22,32 @@ AgentCase, AgentSkill, AtomicFact, + Decision, Episode, Foresight, KnowledgeTopic, + Principle, UserProfile, agent_case_repo, agent_skill_repo, atomic_fact_repo, + decision_repo, episode_repo, foresight_repo, knowledge_topic_repo, + principle_repo, user_profile_repo, ) from everos.infra.persistence.markdown import ( AgentCaseDailyFrontmatter, AgentSkillFrontmatter, AtomicFactDailyFrontmatter, + DecisionDailyFrontmatter, EpisodeDailyFrontmatter, ForesightDailyFrontmatter, KnowledgeDocumentFrontmatter, KnowledgeTopicFrontmatter, + PrincipleFrontmatter, UserProfileFrontmatter, ) @@ -49,12 +55,14 @@ AgentCaseHandler, AgentSkillHandler, AtomicFactHandler, + DecisionHandler, EpisodeHandler, ForesightHandler, Handler, HandlerDeps, KnowledgeDocumentHandler, KnowledgeTopicHandler, + PrincipleHandler, UserProfileHandler, ) @@ -114,6 +122,13 @@ def matches(self, rel_md_path: str) -> bool: lance_repo=foresight_repo, handler_factory=ForesightHandler, ), + KindSpec( + name="decision", + frontmatter_schema=DecisionDailyFrontmatter, + lance_schema=Decision, + lance_repo=decision_repo, + handler_factory=DecisionHandler, + ), KindSpec( name="agent_case", frontmatter_schema=AgentCaseDailyFrontmatter, @@ -135,6 +150,13 @@ def matches(self, rel_md_path: str) -> bool: lance_repo=user_profile_repo, handler_factory=UserProfileHandler, ), + KindSpec( + name="principle", + frontmatter_schema=PrincipleFrontmatter, + lance_schema=Principle, + lance_repo=principle_repo, + handler_factory=PrincipleHandler, + ), KindSpec( name="knowledge_document", frontmatter_schema=KnowledgeDocumentFrontmatter, diff --git a/src/everos/memory/events.py b/src/everos/memory/events.py index 6a58972d2..f017ef14a 100644 --- a/src/everos/memory/events.py +++ b/src/everos/memory/events.py @@ -3,6 +3,7 @@ from __future__ import annotations from everalgo.types import MemCell +from pydantic import Field from everos.infra.ome.events import BaseEvent @@ -10,10 +11,11 @@ class UserPipelineStarted(BaseEvent): """Fired at the start of :class:`UserMemoryPipeline.run`, once per cell. - Hot-path emit, so atomic_fact / foresight / clustering strategies can - start in parallel with the in-pipeline Episode extraction. Carries the - algo-side ``MemCell`` so crash recovery has the full payload (OME - serialises events to JSON via Pydantic v2 nested-model handling). + Hot-path emit, so atomic_fact / foresight / decision / clustering + strategies can start in parallel with the in-pipeline Episode + extraction. Carries the algo-side ``MemCell`` so crash recovery has + the full payload (OME serialises events to JSON via Pydantic v2 + nested-model handling). """ memcell_id: str @@ -62,6 +64,35 @@ class EpisodeExtracted(BaseEvent): source: str = "pipeline" +class DecisionExtracted(BaseEvent): + """Fired once per Decision after ``extract_decision`` writes its md. + + Carries ``decision_text`` so downstream clustering (gate 6a) can work + without racing cascade / polling LanceDB. ``decision_timestamp_ms`` + stamps algo-side ``Cluster.last_ts``. One memcell can produce multiple + decisions, each fanned out once per user sender, so this event fires + per written entry, not per-memcell. + + ``source`` is ``"pipeline"`` for extractor output. Reflection later + emits the same event with ``source="reflection"`` so clustering can + skip a re-cluster of an already-merged decision. + """ + + memcell_id: str + decision_entry_id: str + title: str + decision_text: str + reason: str + impact: str | None = None + tags: list[str] = Field(default_factory=list) + decision_timestamp_ms: int + owner_id: str + session_id: str | None = None + app_id: str = "default" + project_id: str = "default" + source: str = "pipeline" + + class AgentCaseExtracted(BaseEvent): """Fired by ``extract_agent_case`` after the AgentCase md is written. @@ -107,6 +138,30 @@ class ProfileClusterUpdated(BaseEvent): project_id: str = "default" +class DecisionClusterUpdated(BaseEvent): + """Fired after the decision cluster strategy has merged a new + decision into a sqlite cluster (``kind=decision``). + + Snapshot fields (title / body / reason / impact / tags / timestamp) + ride along so a later principle-extraction strategy can consume the + triggering row without racing cascade. Defaults keep OME run_record + back-compat if an older payload is still queued. + """ + + memcell_id: str + decision_entry_id: str + cluster_id: str + owner_id: str + app_id: str = "default" + project_id: str = "default" + title: str = "" + decision_text: str = "" + reason: str = "" + impact: str | None = None + tags: list[str] = Field(default_factory=list) + decision_timestamp_ms: int = 0 + + class SkillClusterUpdated(BaseEvent): """Fired after the agent-case cluster strategy has merged a new case into a cluster. diff --git a/src/everos/memory/get/__init__.py b/src/everos/memory/get/__init__.py index 53274ef53..db39ecc78 100644 --- a/src/everos/memory/get/__init__.py +++ b/src/everos/memory/get/__init__.py @@ -15,6 +15,7 @@ GetAgentCaseItem, GetAgentSkillItem, GetData, + GetDecisionItem, GetEpisodeItem, GetManager, GetMemoryType, @@ -28,6 +29,7 @@ from .dto import GetAgentCaseItem as GetAgentCaseItem from .dto import GetAgentSkillItem as GetAgentSkillItem from .dto import GetData as GetData +from .dto import GetDecisionItem as GetDecisionItem from .dto import GetEpisodeItem as GetEpisodeItem from .dto import GetMemoryType as GetMemoryType from .dto import GetProfileItem as GetProfileItem @@ -40,6 +42,7 @@ "GetAgentCaseItem", "GetAgentSkillItem", "GetData", + "GetDecisionItem", "GetEpisodeItem", "GetManager", "GetMemoryType", diff --git a/src/everos/memory/get/dto.py b/src/everos/memory/get/dto.py index df9804c63..43fb04d4c 100644 --- a/src/everos/memory/get/dto.py +++ b/src/everos/memory/get/dto.py @@ -6,10 +6,10 @@ * ``owner_type`` × ``memory_type`` are strictly paired: - - ``user`` → ``episode`` | ``profile`` + - ``user`` → ``episode`` | ``decision`` | ``profile`` - ``agent`` → ``agent_case`` | ``agent_skill`` -* ``GetData`` always contains four kind arrays for symmetry with +* ``GetData`` always contains kind arrays for symmetry with ``/search``; only the requested kind is populated. ``total_count`` is the predicate's true match count; ``count`` is the page size actually returned. @@ -33,18 +33,20 @@ class GetMemoryType(StrEnum): - """The four kinds enumerated by ``/get``. + """The kinds enumerated by ``/get``. - ``episode`` and ``profile`` are user-owned; ``agent_case`` and - ``agent_skill`` are agent-owned. Cross-pairs are rejected by - :meth:`GetRequest._validate_owner_memory_type_pair`. + ``episode``, ``decision``, and ``profile`` are user-owned; + ``agent_case`` and ``agent_skill`` are agent-owned. Cross-pairs + are rejected by :meth:`GetRequest._validate_owner_memory_type_pair`. + There is no ``principle`` value — Principle is Meta Memory and is + not listed here. - Naming note: all four values use the bare kind name (no - ``_memory`` suffix) and match the LanceDB table name + everalgo - type name for that kind. + Naming note: values use the bare kind name (no ``_memory`` suffix) + and match the LanceDB table name for that kind. """ EPISODE = "episode" + DECISION = "decision" PROFILE = "profile" AGENT_CASE = "agent_case" AGENT_SKILL = "agent_skill" @@ -67,8 +69,9 @@ class GetRequest(BaseModel): user_id: str | None = Field(default=None, min_length=1) agent_id: str | None = Field(default=None, min_length=1) - """Memory owner — provide ``user_id`` for ``episode`` / ``profile`` or - ``agent_id`` for ``agent_case`` / ``agent_skill``; exactly one must be set.""" + """Memory owner — provide ``user_id`` for ``episode`` / ``decision`` / + ``profile`` or ``agent_id`` for ``agent_case`` / ``agent_skill``; + exactly one must be set.""" app_id: str = "default" project_id: str = "default" """App / project scope (default ``"default"``). Pinned into the query @@ -96,7 +99,11 @@ def _validate_user_xor_agent(self) -> Self: def _validate_owner_memory_type_pair(self) -> Self: # Runs after the xor validator (declaration order), so ``owner_type`` # is well-defined here. - user_kinds = {GetMemoryType.EPISODE, GetMemoryType.PROFILE} + user_kinds = { + GetMemoryType.EPISODE, + GetMemoryType.DECISION, + GetMemoryType.PROFILE, + } agent_kinds = {GetMemoryType.AGENT_CASE, GetMemoryType.AGENT_SKILL} if self.owner_type == "user" and self.memory_type not in user_kinds: raise ValueError( @@ -142,6 +149,24 @@ class GetEpisodeItem(BaseModel): type: Literal["Conversation"] +class GetDecisionItem(BaseModel): + """Decision listing item — always user-scoped. No score (unranked).""" + + model_config = ConfigDict(extra="forbid") + + id: str + user_id: str | None + app_id: str = "default" + project_id: str = "default" + session_id: str | None = None + timestamp: _dt.datetime + title: str + decision: str + reason: str + impact: str | None = None + tags: list[str] = Field(default_factory=list) + + class GetProfileItem(BaseModel): """Owner profile — at most one per response, only for user owners.""" @@ -194,7 +219,7 @@ class GetAgentSkillItem(BaseModel): class GetData(BaseModel): """Body of ``response.data``. - All four arrays are always present so client code can iterate + All kind arrays are always present so client code can iterate without branching on ``memory_type``; the route populates exactly one. """ @@ -202,6 +227,7 @@ class GetData(BaseModel): model_config = ConfigDict(extra="forbid") episodes: list[GetEpisodeItem] = Field(default_factory=list) + decisions: list[GetDecisionItem] = Field(default_factory=list) profiles: list[GetProfileItem] = Field(default_factory=list) agent_cases: list[GetAgentCaseItem] = Field(default_factory=list) agent_skills: list[GetAgentSkillItem] = Field(default_factory=list) diff --git a/src/everos/memory/get/manager.py b/src/everos/memory/get/manager.py index c42edd7ea..a071af037 100644 --- a/src/everos/memory/get/manager.py +++ b/src/everos/memory/get/manager.py @@ -4,15 +4,17 @@ :class:`GetRequest`): * ``user`` + ``episode`` → ``data.episodes`` +* ``user`` + ``decision`` → ``data.decisions`` * ``user`` + ``profile`` → ``data.profiles`` (one-row KV fetch from the ``user_profile`` table; at most one item) * ``agent`` + ``agent_case`` → ``data.agent_cases`` * ``agent`` + ``agent_skill`` → ``data.agent_skills`` Reads only — never writes. Filters are compiled through -:func:`compile_filters_for_get` so the column allow-list stays -shared with :mod:`memory.search`. Pagination + in-memory sort -runs through :meth:`LanceRepoBase.find_where_paginated`. +:func:`compile_filters_for_get` (or :func:`compile_filters_for_decision` +when ``memory_type=decision``, which strips ``sender_id``) so the +column allow-list stays shared with :mod:`memory.search`. Pagination ++ in-memory sort runs through :meth:`LanceRepoBase.find_where_paginated`. """ from __future__ import annotations @@ -23,11 +25,13 @@ from everos.component.utils.datetime import to_display_tz from everos.core.context import resolve_request_id from everos.core.observability.logging import get_logger +from everos.memory.search import compile_filters_for_decision from .dto import ( GetAgentCaseItem, GetAgentSkillItem, GetData, + GetDecisionItem, GetEpisodeItem, GetMemoryType, GetProfileItem, @@ -41,6 +45,7 @@ from everos.infra.persistence.lancedb import ( AgentCase, AgentSkill, + Decision, Episode, UserProfile, ) @@ -56,11 +61,13 @@ def __init__( self, *, episode_repo: LanceRepoBase[Episode], + decision_repo: LanceRepoBase[Decision], agent_case_repo: LanceRepoBase[AgentCase], agent_skill_repo: LanceRepoBase[AgentSkill], user_profile_repo: LanceRepoBase[UserProfile], ) -> None: self._ep = episode_repo + self._decision = decision_repo self._case = agent_case_repo self._skill = agent_skill_repo self._profile = user_profile_repo @@ -70,7 +77,12 @@ def __init__( async def get(self, req: GetRequest) -> GetResponse: request_id = resolve_request_id() descending = req.sort_order == "desc" - where = compile_filters_for_get( + compile = ( + compile_filters_for_decision + if req.memory_type == GetMemoryType.DECISION + else compile_filters_for_get + ) + where = compile( req.filters, owner_id=req.owner_id, owner_type=req.owner_type, @@ -93,6 +105,20 @@ async def get(self, req: GetRequest) -> GetResponse: total_count=total, count=len(items), ) + case GetMemoryType.DECISION: + rows, total = await self._decision.find_where_paginated( + where, + sort_by=req.sort_by, + descending=descending, + page=req.page, + page_size=req.page_size, + ) + items = [self._shape_decision(r) for r in rows] + data = GetData( + decisions=items, + total_count=total, + count=len(items), + ) case GetMemoryType.PROFILE: profiles = await self._fetch_profile(req.owner_id) data = GetData( @@ -153,6 +179,22 @@ def _shape_episode(row: Episode) -> GetEpisodeItem: type="Conversation", ) + @staticmethod + def _shape_decision(row: Decision) -> GetDecisionItem: + return GetDecisionItem( + id=row.id, + user_id=row.owner_id, + app_id=row.app_id, + project_id=row.project_id, + session_id=row.session_id, + timestamp=to_display_tz(row.timestamp), + title=row.title, + decision=row.decision, + reason=row.reason, + impact=row.impact, + tags=list(row.tags), + ) + @staticmethod def _shape_agent_case(row: AgentCase) -> GetAgentCaseItem: return GetAgentCaseItem( diff --git a/src/everos/memory/models.py b/src/everos/memory/models.py index b83f20b31..cde3e4fb0 100644 --- a/src/everos/memory/models.py +++ b/src/everos/memory/models.py @@ -16,6 +16,7 @@ from everalgo.types import AgentCase as AlgoAgentCase from everalgo.types import AtomicFact as AlgoAtomicFact from everalgo.types import ChatMessage as AlgoMessage +from everalgo.types import Decision as AlgoDecision from everalgo.types import Episode as AlgoEpisode from everalgo.types import Foresight as AlgoForesight from everalgo.types import MemCell as MemCell @@ -152,6 +153,65 @@ def from_algo( return cls.model_validate(data) +class Decision(BaseModel): + """Domain Decision — algo-emitted business fields + everos context. + + Composed (not inherited) from :class:`everalgo.types.Decision`. everos + keeps the semantic fields algo emits (``title`` / ``decision`` / + ``reason`` / ``impact`` / ``tags`` / ``timestamp``) and adds + engineering context (``session_id`` / ``parent_id``). The global id is + derived later by cascade from ``_``. + + ``parent_id`` is the source memcell id, same as :class:`Episode`. Algo + Decision has no ``parent_id``; everos fills it from the memcell currently + being processed. + + No ``sender_ids``: a decision is a committed trade-off about its + ``owner_id``, not a narrative about the conversation as a whole. + """ + + owner_id: str + title: str + decision: str + reason: str + impact: str | None = None + tags: list[str] = Field(default_factory=list) + timestamp: int + + # everos engineering metadata. + session_id: str | None = None + parent_id: str + + model_config = ConfigDict(extra="allow") + + @classmethod + def from_algo( + cls, + algo_decision: AlgoDecision, + *, + owner_id: str, + session_id: str | None, + parent_id: str, + ) -> Decision: + """Build a domain Decision from an algo Decision plus engineering context. + + ``owner_id`` is caller-supplied so the same generic algo Decision + (produced once per MemCell, ``owner_id=None``) can fan out to one md + per user sender. Any ``owner_id`` algo's model might carry is dropped — + the caller's context is authoritative. + + ``parent_id`` is required for the same reason: the caller always knows + the source memcell id. Anything algo's model carries via + ``extra='allow'`` is dropped in favour of the caller-supplied value. + ``app_id`` / ``project_id`` stay on writer / cascade scope, not here. + """ + data = algo_decision.model_dump(exclude={"parent_id", "owner_id"}) + data["owner_id"] = owner_id + data["session_id"] = session_id + data["parent_id"] = parent_id + return cls.model_validate(data) + + class AtomicFact(BaseModel): """Domain AtomicFact — algo-emitted business fields + everos context. @@ -328,11 +388,13 @@ def from_algo( "AgentCase", "AlgoAgentCase", "AlgoAtomicFact", + "AlgoDecision", "AlgoEpisode", "AlgoForesight", "AlgoMessage", "AtomicFact", "CanonicalMessage", + "Decision", "Episode", "Foresight", "IngestResult", diff --git a/src/everos/memory/reflection/__init__.py b/src/everos/memory/reflection/__init__.py index aa46fa13d..c2fa1ccc1 100644 --- a/src/everos/memory/reflection/__init__.py +++ b/src/everos/memory/reflection/__init__.py @@ -1,14 +1,24 @@ """Reflection — offline memory consolidation. -Merges fragmented cluster members into higher-quality episodes, re- -extracts atomic facts, and deprecates the originals. +Episode path: merge fragmented ``kind=user_memory`` cluster members, +re-extract atomic facts, deprecate originals. + +Decision path: merge fragmented ``kind=decision`` cluster members into +one Decision, emit ``DecisionExtracted(source="reflection")``, deprecate +originals. No atomic-fact re-extract. External usage: - from everos.memory.reflection import ReflectionOrchestrator + from everos.memory.reflection import ( + DecisionReflectionOrchestrator, + ReflectionOrchestrator, + ) """ from __future__ import annotations +from .decision_orchestrator import ( + DecisionReflectionOrchestrator as DecisionReflectionOrchestrator, +) from .orchestrator import ReflectionOrchestrator as ReflectionOrchestrator -__all__ = ["ReflectionOrchestrator"] +__all__ = ["DecisionReflectionOrchestrator", "ReflectionOrchestrator"] diff --git a/src/everos/memory/reflection/decision_orchestrator.py b/src/everos/memory/reflection/decision_orchestrator.py new file mode 100644 index 000000000..23ead5410 --- /dev/null +++ b/src/everos/memory/reflection/decision_orchestrator.py @@ -0,0 +1,725 @@ +"""DecisionReflectionOrchestrator — Select -> Merge -> Write -> Deprecate. + +Consolidates fragmented sqlite ``kind=decision`` cluster members into +one merged Decision per cluster. The merged entry is written to +markdown (``parent_type=cluster``), ``DecisionExtracted(source= +"reflection")`` is emitted so clustering can skip it, and the originals +are deprecated in md frontmatter and Lance ``deprecated_by``. + +Copied from :class:`ReflectionOrchestrator` (episode). Do **not** edit +that file — Decision reflection is a sibling cycle, not a flag on the +episode path. There is no atomic-fact re-extract: Decision is not an +episode, and ``wait_for_event`` would hang because no strategy consumes +``source="reflection"`` (``trigger_decision_clustering`` applies only +to ``source="pipeline"``). +""" + +from __future__ import annotations + +import asyncio +import datetime as _dt +import json +import uuid +from collections import defaultdict +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from everalgo.types import Decision as AlgoDecision + from everalgo.user_memory import DecisionReflector + + from everos.component.embedding import EmbeddingProvider + from everos.infra.persistence.markdown import DecisionWriter + +import numpy as np + +from everos.component.utils.datetime import from_timestamp, to_iso_format +from everos.core.errors import AppError +from everos.core.observability.logging import get_logger +from everos.core.observability.tracing import memory_span +from everos.core.persistence import MemoryRoot +from everos.infra.ome.context import StrategyContext +from everos.memory._partition_locks import get_partition_lock +from everos.memory.events import DecisionExtracted + +logger = get_logger(__name__) + +_MAX_CLUSTERS_PER_RUN = 10 + + +def _escape_sql(value: str) -> str: + """Escape single quotes for LanceDB SQL-like ``where`` predicates.""" + return value.replace("'", "''") + + +class DecisionReflectionOrchestrator: + """Run one Decision Reflection cycle for a single owner scope. + + Args: + cluster_repo: SQLite cluster repository. + decision_store: LanceDB decision repository (read + update). + decision_writer: Markdown daily-log writer for decisions. + report_repo: SQLite reflection report repository. + reflector: Algorithm-side DecisionReflector (areflect). + embedder: Embedding provider for centroid recomputation. + """ + + def __init__( + self, + *, + cluster_repo: Any, + decision_store: Any, + decision_writer: DecisionWriter, + report_repo: Any, + reflector: DecisionReflector, + embedder: EmbeddingProvider, + ) -> None: + self._cluster_repo = cluster_repo + self._decision_store = decision_store + self._decision_writer = decision_writer + self._report_repo = report_repo + self._reflector = reflector + self._embedder = embedder + + async def run( + self, + *, + ctx: StrategyContext, + owner_id: str, + owner_type: str = "user", + kind: str = "decision", + app_id: str = "default", + project_id: str = "default", + ) -> list[object]: + """Run one Decision Reflection cycle for a single owner scope.""" + candidates = await self._select_candidates( + owner_id=owner_id, + kind=kind, + app_id=app_id, + project_id=project_id, + ) + logger.info( + "decision_reflection_candidates_selected", + owner_id=owner_id, + candidate_count=len(candidates), + ) + if not candidates: + return [] + + reports: list[object] = [] + skip_count = 0 + for cluster_id in candidates: + report = await self._process_cluster_safely( + ctx=ctx, + cluster_id=cluster_id, + owner_id=owner_id, + owner_type=owner_type, + app_id=app_id, + project_id=project_id, + ) + if report is not None: + reports.append(report) + else: + skip_count += 1 + + logger.info( + "decision_reflection_cycle_completed", + owner_id=owner_id, + success_count=len(reports), + skip_count=skip_count, + ) + return reports + + async def _process_cluster_safely( + self, + *, + ctx: StrategyContext, + cluster_id: str, + owner_id: str, + owner_type: str, + app_id: str, + project_id: str, + ) -> object | None: + try: + return await self._process_cluster( + ctx=ctx, + cluster_id=cluster_id, + owner_id=owner_id, + owner_type=owner_type, + app_id=app_id, + project_id=project_id, + ) + except AppError: + logger.warning( + "decision_reflection_cluster_skipped", + cluster_id=cluster_id, + exc_info=True, + ) + return None + except Exception: + logger.error( + "decision_reflection_cluster_unexpected_error", + cluster_id=cluster_id, + exc_info=True, + ) + return None + + async def _select_candidates( + self, + *, + owner_id: str, + kind: str, + app_id: str, + project_id: str, + ) -> list[str]: + reflected = await self._report_repo.list_reflected_cluster_ids( + owner_id, app_id, project_id + ) + clusters = await self._cluster_repo.list_ids_and_member_counts( + owner_id, kind, app_id=app_id, project_id=project_id + ) + count_map = dict(clusters) + candidates = [ + cid + for cid, count in clusters + if (cid not in reflected and count >= 2) or (cid in reflected and count > 1) + ] + candidates.sort(key=lambda cid: count_map[cid], reverse=True) + return candidates[:_MAX_CLUSTERS_PER_RUN] + + async def _process_cluster( + self, + *, + ctx: StrategyContext, + cluster_id: str, + owner_id: str, + owner_type: str, + app_id: str, + project_id: str, + ) -> object | None: + await self._detect_orphans(cluster_id, owner_id, app_id, project_id) + + scope = dict(owner_id=owner_id, app_id=app_id, project_id=project_id) + members, decisions = await self._load_cluster_decisions( + cluster_id=cluster_id, **scope + ) + if not members or not decisions: + return None + + mode, algo_result = await self._reflect_cluster( + decisions=decisions, + owner_id=owner_id, + ) + if algo_result is None: + return None + + merged_entry_id = await self._write_and_emit( + ctx=ctx, + cluster_id=cluster_id, + **scope, + algo_result=algo_result, + decisions=decisions, + mode=mode, + members=members, + ) + if merged_entry_id is None: + return None + + return await self._deprecate( + ctx=ctx, + cluster_id=cluster_id, + owner_type=owner_type, + **scope, + original_members=members, + merged_entry_id=merged_entry_id, + algo_result=algo_result, + mode=mode, + decisions=decisions, + ) + + async def _reflect_cluster( + self, + *, + decisions: list[Any], + owner_id: str, + ) -> tuple[str, AlgoDecision | None]: + merged_entry_ids = [d.entry_id for d in decisions if d.parent_type == "cluster"] + is_update = bool(merged_entry_ids) + mode = "update" if is_update else "init" + algo_result = await self._call_reflector( + decisions=decisions, + merged_entry_ids=merged_entry_ids, + is_update=is_update, + owner_id=owner_id, + ) + return mode, algo_result + + async def _load_cluster_decisions( + self, + *, + cluster_id: str, + owner_id: str, + app_id: str, + project_id: str, + ) -> tuple[list[tuple[str, str]], list[Any]]: + members = await self._cluster_repo.get_members_with_type(cluster_id) + if not members: + return [], [] + member_ids = [mid for mid, _ in members] + rows = await self._decision_store.find_by_owner_entries( + owner_id, + member_ids, + app_id=app_id, + project_id=project_id, + ) + rows.sort(key=lambda d: d.timestamp) + return members, rows + + async def _write_and_emit( + self, + *, + ctx: StrategyContext, + cluster_id: str, + owner_id: str, + app_id: str, + project_id: str, + algo_result: AlgoDecision, + decisions: list[Any], + mode: str, + members: list[tuple[str, str]], + ) -> str | None: + last_ts = max(row.timestamp for row in decisions) + merged_entry_id = await self._write_merged_decision( + cluster_id=cluster_id, + owner_id=owner_id, + app_id=app_id, + project_id=project_id, + algo_result=algo_result, + last_ts=last_ts, + ) + logger.info( + "decision_reflection_merged", + cluster_id=cluster_id, + mode=mode, + source_count=len(members), + merged_entry_id=merged_entry_id, + ) + event = DecisionExtracted( + memcell_id=merged_entry_id, + decision_entry_id=merged_entry_id, + title=algo_result.title, + decision_text=algo_result.decision, + reason=algo_result.reason, + impact=algo_result.impact, + tags=list(algo_result.tags), + decision_timestamp_ms=_ts_to_ms(last_ts), + owner_id=owner_id, + session_id=None, + app_id=app_id, + project_id=project_id, + source="reflection", + ) + await ctx.emit(event) + # No wait_for_event: clustering applies_to pipeline only, and + # there is no atomic-fact re-extract for a merged Decision. + return merged_entry_id + + async def _write_merged_decision( + self, + *, + cluster_id: str, + owner_id: str, + app_id: str, + project_id: str, + algo_result: AlgoDecision, + last_ts: object, + ) -> str: + last_ts_iso = to_iso_format(from_timestamp(_ts_to_ms(last_ts))) + if last_ts_iso is None: + raise ValueError("to_iso_format returned None for valid timestamp") + inline, sections = _merged_decision_to_entry_body( + algo_result, cluster_id, owner_id, last_ts_iso + ) + entry_ids = await self._decision_writer.append_entries( + owner_id, + [(inline, sections)], + app_id=app_id, + project_id=project_id, + ) + return entry_ids[0].format() + + async def _detect_orphans( + self, + cluster_id: str, + owner_id: str, + app_id: str, + project_id: str, + ) -> None: + where = ( + f"parent_type = 'cluster' AND parent_id = '{_escape_sql(cluster_id)}' " + f"AND deprecated_by IS NULL " + f"AND owner_id = '{_escape_sql(owner_id)}' " + f"AND app_id = '{_escape_sql(app_id)}' " + f"AND project_id = '{_escape_sql(project_id)}'" + ) + orphans = await self._decision_store.find_where(where, limit=10) + if orphans: + logger.warning( + "decision_reflection_orphan_detected", + cluster_id=cluster_id, + orphan_entry_ids=[o.entry_id for o in orphans], + ) + + async def _call_reflector( + self, + *, + decisions: list[Any], + merged_entry_ids: list[str], + is_update: bool, + owner_id: str, + ) -> AlgoDecision | None: + algo_decisions = _to_algo_decisions(decisions) + try: + with memory_span( + "everos.reflect.decision_consolidate", + observation_type="generation", + metadata={"owner_id": owner_id, "is_update": is_update}, + ): + if is_update: + return await self._reflect_update( + algo_decisions=algo_decisions, + decisions=decisions, + merged_entry_ids=merged_entry_ids, + ) + return await self._reflector.areflect(algo_decisions) + except AppError: + logger.warning( + "decision_reflection_reflector_failed", + owner_id=owner_id, + exc_info=True, + ) + return None + except Exception: + logger.error( + "decision_reflection_reflector_unexpected_error", + owner_id=owner_id, + exc_info=True, + ) + return None + + async def _reflect_update( + self, + *, + algo_decisions: list[AlgoDecision], + decisions: list[Any], + merged_entry_ids: list[str], + ) -> AlgoDecision | None: + merged_set = set(merged_entry_ids) + old_algo = [ + ad + for ad, d in zip(algo_decisions, decisions, strict=True) + if d.entry_id in merged_set + ] + new_algo = [ + ad + for ad, d in zip(algo_decisions, decisions, strict=True) + if d.entry_id not in merged_set + ] + if not old_algo: + return None + return await self._reflector.areflect(new_algo, old_decision=old_algo[0]) + + async def _deprecate( + self, + *, + ctx: StrategyContext, + cluster_id: str, + owner_id: str, + owner_type: str, + app_id: str, + project_id: str, + original_members: list[tuple[str, str]], + merged_entry_id: str, + algo_result: AlgoDecision, + mode: str, + decisions: list[Any], + ) -> object | None: + """Deprecate originals and update cluster membership. + + ``ctx`` / ``owner_type`` are unused here (kept for signature + parity with :class:`ReflectionOrchestrator`). + """ + partition = f"{app_id}:{project_id}:{cluster_id}" + try: + async with get_partition_lock("decision_reflection_deprecate", partition): + return await self._execute_deprecation( + cluster_id=cluster_id, + owner_id=owner_id, + app_id=app_id, + project_id=project_id, + original_members=original_members, + merged_entry_id=merged_entry_id, + algo_result=algo_result, + mode=mode, + decisions=decisions, + ) + except AppError: + logger.warning( + "decision_reflection_deprecate_failed", + cluster_id=cluster_id, + exc_info=True, + ) + return None + except Exception: + logger.error( + "decision_reflection_deprecate_unexpected_error", + cluster_id=cluster_id, + exc_info=True, + ) + return None + + async def _execute_deprecation( + self, + *, + cluster_id: str, + owner_id: str, + app_id: str, + project_id: str, + original_members: list[tuple[str, str]], + merged_entry_id: str, + algo_result: AlgoDecision, + mode: str, + decisions: list[Any], + ) -> object | None: + to_deprecate = await self._resolve_deprecation_targets( + cluster_id=cluster_id, + original_members=original_members, + ) + if not to_deprecate: + return None + + dep_count = await self._apply_deprecation_writes( + decisions=decisions, + to_deprecate=to_deprecate, + owner_id=owner_id, + app_id=app_id, + project_id=project_id, + merged_entry_id=merged_entry_id, + ) + await self._update_cluster_after_merge( + cluster_id=cluster_id, + to_deprecate=to_deprecate, + merged_entry_id=merged_entry_id, + algo_result=algo_result, + decisions=decisions, + ) + report = await self._create_reflection_report( + cluster_id=cluster_id, + owner_id=owner_id, + app_id=app_id, + project_id=project_id, + mode=mode, + original_members=original_members, + to_deprecate=to_deprecate, + merged_entry_id=merged_entry_id, + ) + logger.info( + "decision_reflection_deprecated", + cluster_id=cluster_id, + deprecated_decision_count=dep_count, + ) + return report + + async def _apply_deprecation_writes( + self, + *, + decisions: list[Any], + to_deprecate: set[str], + owner_id: str, + app_id: str, + project_id: str, + merged_entry_id: str, + ) -> int: + await self._patch_md_frontmatter( + decisions=decisions, + to_deprecate=to_deprecate, + merged_entry_id=merged_entry_id, + ) + return await self._deprecate_lance_decisions( + entry_ids=to_deprecate, + owner_id=owner_id, + app_id=app_id, + project_id=project_id, + merged_entry_id=merged_entry_id, + ) + + async def _resolve_deprecation_targets( + self, + *, + cluster_id: str, + original_members: list[tuple[str, str]], + ) -> set[str]: + current_members = await self._cluster_repo.get_members_with_type(cluster_id) + current_ids = {mid for mid, _ in current_members} + original_ids = {mid for mid, _ in original_members} + return original_ids & current_ids + + async def _deprecate_lance_decisions( + self, + *, + entry_ids: set[str], + owner_id: str, + app_id: str, + project_id: str, + merged_entry_id: str, + ) -> int: + coros: list[Any] = [ + self._decision_store.update( + {"deprecated_by": merged_entry_id}, + where=( + f"entry_id = '{_escape_sql(eid)}' " + f"AND owner_id = '{_escape_sql(owner_id)}' " + f"AND app_id = '{_escape_sql(app_id)}' " + f"AND project_id = '{_escape_sql(project_id)}'" + ), + ) + for eid in entry_ids + ] + if coros: + await asyncio.gather(*coros) + return len(coros) + + async def _update_cluster_after_merge( + self, + *, + cluster_id: str, + to_deprecate: set[str], + merged_entry_id: str, + algo_result: AlgoDecision, + decisions: list[Any], + ) -> None: + await self._cluster_repo.remove_members(cluster_id, to_deprecate) + await self._cluster_repo.add_member(cluster_id, merged_entry_id, "decision") + + centroid = await self._embedder.embed(algo_result.decision) + centroid_blob = np.asarray(centroid, dtype=np.float32).tobytes() + last_ts_ms = _ts_to_ms(max(row.timestamp for row in decisions)) + await self._cluster_repo.update_metadata( + cluster_id, + centroid_blob=centroid_blob, + count=1, + last_ts_ms=last_ts_ms, + preview_json=json.dumps([algo_result.decision[:200]], ensure_ascii=False), + ) + + async def _create_reflection_report( + self, + *, + cluster_id: str, + owner_id: str, + app_id: str, + project_id: str, + mode: str, + original_members: list[tuple[str, str]], + to_deprecate: set[str], + merged_entry_id: str, + ) -> object: + from everos.infra.persistence.sqlite import ReflectionReport + + source_members_json = json.dumps( + [ + {"member_id": mid, "member_type": mtype} + for mid, mtype in original_members + if mid in to_deprecate + ], + ensure_ascii=False, + ) + report = ReflectionReport( + id=uuid.uuid4().hex, + cluster_id=cluster_id, + owner_id=owner_id, + app_id=app_id, + project_id=project_id, + mode=mode, + source_members=source_members_json, + source_count=len(to_deprecate), + merged_entry_id=merged_entry_id, + deprecated_fact_count=0, + ) + await self._report_repo.create(report) + return report + + async def _patch_md_frontmatter( + self, + *, + decisions: list[Any], + to_deprecate: set[str], + merged_entry_id: str, + ) -> None: + path_to_entries: dict[str, dict[str, str]] = defaultdict(dict) + for row in decisions: + is_deprecated = ( + row.parent_id in to_deprecate or row.entry_id in to_deprecate + ) + if is_deprecated and row.md_path: + path_to_entries[row.md_path][row.entry_id] = merged_entry_id + + root = MemoryRoot.resolve().root + for md_path, deprecated_map in path_to_entries.items(): + await self._decision_writer.patch_frontmatter( + root / md_path, + {"deprecated_entries": deprecated_map}, + ) + + +def _to_algo_decisions(rows: list[Any]) -> list[AlgoDecision]: + from everalgo.types import Decision as AlgoDecision + + return [ + AlgoDecision( + owner_id=row.owner_id, + title=row.title, + decision=row.decision, + reason=row.reason, + impact=row.impact, + tags=list(row.tags), + timestamp=_ts_to_ms(row.timestamp), + ) + for row in rows + ] + + +def _merged_decision_to_entry_body( + algo_result: AlgoDecision, + cluster_id: str, + owner_id: str, + timestamp_iso: str, +) -> tuple[dict[str, object], dict[str, str]]: + """Build ``(inline, sections)`` for a merged decision md entry. + + ``session_id`` is omitted — a cluster merge has no conversation + session. ``parent_type`` is ``cluster``. + """ + inline: dict[str, object] = { + "owner_id": owner_id, + "timestamp": timestamp_iso, + "parent_type": "cluster", + "parent_id": cluster_id, + "tags": list(algo_result.tags), + } + sections: dict[str, str] = { + "Title": algo_result.title, + "Decision": algo_result.decision, + "Reason": algo_result.reason, + } + if algo_result.impact: + sections["Impact"] = algo_result.impact + return inline, sections + + +def _ts_to_ms(ts: object) -> int: + """Coerce a Lance datetime or algo-ms int to milliseconds.""" + if isinstance(ts, _dt.datetime): + return int(ts.timestamp() * 1000) + if isinstance(ts, (int, float)): + return int(ts) + raise TypeError(f"unexpected timestamp type: {type(ts)}") diff --git a/src/everos/memory/reflection/orchestrator.py b/src/everos/memory/reflection/orchestrator.py index 547adc643..e593e93bf 100644 --- a/src/everos/memory/reflection/orchestrator.py +++ b/src/everos/memory/reflection/orchestrator.py @@ -1047,6 +1047,7 @@ def _to_algo_episodes(episodes: list[Any]) -> list[AlgoEpisode]: owner_id=e.owner_id, episode=e.episode, subject=e.subject or "", + summary=e.summary or e.episode, timestamp=_ts_to_ms(e.timestamp), ) for e in episodes diff --git a/src/everos/memory/search/__init__.py b/src/everos/memory/search/__init__.py index da2bef75e..de34833d5 100644 --- a/src/everos/memory/search/__init__.py +++ b/src/everos/memory/search/__init__.py @@ -22,12 +22,15 @@ SearchAgentSkillItem, SearchAtomicFactItem, SearchData, + SearchDecisionItem, SearchEpisodeItem, SearchMethod, + SearchPrincipleItem, SearchProfileItem, SearchRequest, SearchResponse, compile_filters, + compile_filters_for_decision, compile_predicate, ) @@ -41,8 +44,10 @@ from .dto import SearchAgentSkillItem as SearchAgentSkillItem from .dto import SearchAtomicFactItem as SearchAtomicFactItem from .dto import SearchData as SearchData +from .dto import SearchDecisionItem as SearchDecisionItem from .dto import SearchEpisodeItem as SearchEpisodeItem from .dto import SearchMethod as SearchMethod +from .dto import SearchPrincipleItem as SearchPrincipleItem from .dto import SearchProfileItem as SearchProfileItem from .dto import SearchRequest as SearchRequest from .dto import SearchResponse as SearchResponse @@ -50,6 +55,7 @@ from .filters import RESERVED_FIELDS as RESERVED_FIELDS from .filters import FilterError as FilterError from .filters import compile_filters as compile_filters +from .filters import compile_filters_for_decision as compile_filters_for_decision from .filters import compile_predicate as compile_predicate __all__ = [ @@ -61,11 +67,14 @@ "SearchAgentSkillItem", "SearchAtomicFactItem", "SearchData", + "SearchDecisionItem", "SearchEpisodeItem", "SearchMethod", + "SearchPrincipleItem", "SearchProfileItem", "SearchRequest", "SearchResponse", "compile_filters", + "compile_filters_for_decision", "compile_predicate", ] diff --git a/src/everos/memory/search/adapter.py b/src/everos/memory/search/adapter.py index 2d7cef36b..6d9f1329b 100644 --- a/src/everos/memory/search/adapter.py +++ b/src/everos/memory/search/adapter.py @@ -8,6 +8,8 @@ * ``KEYWORD`` / ``VECTOR`` → ``None`` → manager skips ``everalgo.rank``. * ``HYBRID`` → ``"hierarchy"`` (episode / atomic_fact) — heap-expand pipeline (RRF-ordered expansion → LR-calibrated global top-N competition) + or ``"rrf"`` (decision) — sparse + dense fused with + :func:`everalgo.rank.fusion.rrf` (no ``arank``) or ``"vector_anchored"`` (agent_case) — everalgo vector-anchored fusion (alpha=0.7) or ``"skill_hybrid"`` (agent_skill) — custom rrf → cross-encoder rerank → optional verify. @@ -19,7 +21,7 @@ from .dto import SearchMethod -KindName = Literal["episode", "atomic_fact", "agent_case", "agent_skill"] +KindName = Literal["episode", "atomic_fact", "decision", "agent_case", "agent_skill"] def resolve_pipeline( @@ -32,6 +34,8 @@ def resolve_pipeline( the manager runs single-route recall and returns directly". ``"hierarchy"`` routes to the heap-expand episode pipeline in ``memory.search.hierarchy`` (RRF → LR → heap expansion → eviction). + ``"rrf"`` fuses sparse + dense with :func:`everalgo.rank.fusion.rrf` + in the manager — Decision is not an ``arank`` ``memory_type``. ``"vector_anchored"`` routes to ``everalgo.rank.arank`` with vector-anchored fusion (alpha=0.7, saturation_k=5.0) — matches the opensource case retrieval. ``"skill_hybrid"`` routes to the custom skill hybrid orchestrator in @@ -43,6 +47,8 @@ def resolve_pipeline( if method == SearchMethod.HYBRID: if kind in ("episode", "atomic_fact"): return "hierarchy", None + if kind == "decision": + return "rrf", None if kind == "agent_case": return "vector_anchored", None # agent_skill: custom hybrid orchestrator (rrf → cross-encoder → optional diff --git a/src/everos/memory/search/dto.py b/src/everos/memory/search/dto.py index 53d37fa5b..245cc9ceb 100644 --- a/src/everos/memory/search/dto.py +++ b/src/everos/memory/search/dto.py @@ -3,11 +3,15 @@ Contract per the final design: * ``owner_type`` is a hard partition. ``user`` returns ``episodes`` - (and optionally ``profiles``); ``agent`` returns ``agent_cases`` + - ``agent_skills``. The four ``data.*`` arrays always exist; routes not - applicable to the current ``owner_type`` stay as ``[]``. + (and ``decisions``; optionally ``profiles`` / ``principles``); + ``agent`` returns ``agent_cases`` + ``agent_skills``. The ``data.*`` + arrays always exist; routes not applicable to the current + ``owner_type`` stay as ``[]``. * ``atomic_facts`` are **nested** inside :class:`SearchEpisodeItem`, never returned as a top-level array. +* ``principle`` is Meta Memory, not a searchable kind — it is not in + ``kinds``. Callers opt in with ``include_principles`` (KV fetch), + independent of ``include_profile`` and of the episode/decision lanes. * Item-side ``owner_type`` / ``type`` fields are intentionally narrowed to the currently-emitted Literal so callers get a tight schema. Loosen them only when a new emission path (agent episodes, agent profiles) @@ -70,8 +74,8 @@ class SearchRequest(BaseModel): user_id: str | None = Field(default=None, min_length=1) agent_id: str | None = Field(default=None, min_length=1) """Memory owner — provide ``user_id`` for user-memory (episodes / - profiles) or ``agent_id`` for agent-memory (cases / skills); exactly - one must be set.""" + decisions / profiles / principles) or ``agent_id`` for agent-memory + (cases / skills); exactly one must be set.""" app_id: str = "default" project_id: str = "default" """App / project scope (default ``"default"``). Pinned into the LanceDB @@ -89,6 +93,19 @@ class SearchRequest(BaseModel): Only the episode hybrid path consumes it — other methods ignore it. """ include_profile: bool = False + include_principles: bool = False + """When true and ``user_id`` is set, attach the owner's principle + KV rows (``data.principles``). Independent of ``include_profile`` + and of ``kinds``. Ignored for ``agent_id``. Not a HYBRID lane — + principle is not a kind. + """ + kinds: list[Literal["episode", "decision"]] | None = None + """User-partition kind filter. ``None`` searches episode + decision + in parallel. ``["decision"]`` / ``["episode"]`` restrict the lanes. + Rejected when ``agent_id`` is set, when the list is empty, or when + a value outside the Literal is supplied (``principle`` is not a + kind). Independent of ``include_profile`` and ``include_principles``. + """ enable_llm_rerank: bool = Field( default=False, description=( @@ -112,6 +129,16 @@ def _validate_top_k(self) -> SearchRequest: raise ValueError("top_k must be -1 or in 1..100") return self + @model_validator(mode="after") + def _validate_kinds(self) -> SearchRequest: + if self.kinds is None: + return self + if self.agent_id is not None: + raise ValueError("kinds is only valid when user_id is set") + if not self.kinds: + raise ValueError("kinds must be non-empty when provided") + return self + @property def owner_id(self) -> str: """Derived from whichever of ``user_id`` / ``agent_id`` is set. @@ -168,6 +195,26 @@ class SearchEpisodeItem(BaseModel): atomic_facts: list[SearchAtomicFactItem] = Field(default_factory=list) +class SearchDecisionItem(BaseModel): + """Decision hit — always user-scoped. Instance Kind, HYBRID recall.""" + + model_config = ConfigDict(extra="forbid") + + id: str + user_id: str | None + """Owning user (``None`` only on malformed cascade rows).""" + app_id: str = "default" + project_id: str = "default" + session_id: str | None = None + timestamp: _dt.datetime + title: str + decision: str + reason: str + impact: str | None = None + tags: list[str] = Field(default_factory=list) + score: float + + class SearchProfileItem(BaseModel): """Owner profile — at most one per response, only for user owners. @@ -186,6 +233,27 @@ class SearchProfileItem(BaseModel): score: float | None = None +class SearchPrincipleItem(BaseModel): + """One engineering principle — KV fetch, not a ranked kind. + + ``score`` is always ``None`` (no query-relevance). ``id`` is the + Lance PK ``_``. Agent owners never receive + this array. + """ + + model_config = ConfigDict(extra="forbid") + + id: str + user_id: str | None + app_id: str = "default" + project_id: str = "default" + title: str + statement: str + source_entry_ids: list[str] = Field(default_factory=list) + timestamp: _dt.datetime + score: float | None = None + + class SearchAgentCaseItem(BaseModel): """Agent case hit — always agent-scoped.""" @@ -257,7 +325,7 @@ class UnprocessedMessageDTO(BaseModel): class SearchData(BaseModel): """Body of ``response.data``. - All five arrays are always present so client code can iterate without + All arrays are always present so client code can iterate without branching on ``owner_type``. Routes not applicable to the request's owner type stay as ``[]``. ``unprocessed_messages`` is filled only when ``filters.session_id`` is present as a top-level eq scalar — @@ -268,7 +336,11 @@ class SearchData(BaseModel): model_config = ConfigDict(extra="forbid") episodes: list[SearchEpisodeItem] = Field(default_factory=list) + decisions: list[SearchDecisionItem] = Field(default_factory=list) profiles: list[SearchProfileItem] = Field(default_factory=list) + principles: list[SearchPrincipleItem] = Field(default_factory=list) + """KV-fetched principles when ``include_principles=true`` (user + owner); otherwise stays empty. Always present on the wire.""" agent_cases: list[SearchAgentCaseItem] = Field(default_factory=list) agent_skills: list[SearchAgentSkillItem] = Field(default_factory=list) unprocessed_messages: list[UnprocessedMessageDTO] = Field(default_factory=list) diff --git a/src/everos/memory/search/filters.py b/src/everos/memory/search/filters.py index 2dd9aa804..c41cb1c1c 100644 --- a/src/everos/memory/search/filters.py +++ b/src/everos/memory/search/filters.py @@ -108,8 +108,8 @@ def compile_filters( f"app_id = '{_escape_str(app_id)}'", f"project_id = '{_escape_str(project_id)}'", ] - # Only episode / atomic_fact tables carry the ``deprecated_by`` column - # (Reflection V1 marks superseded entries). Agent tables don't have it. + # episode / atomic_fact / decision tables carry ``deprecated_by`` + # (Reflection marks superseded entries). Agent tables don't have it. if owner_type == "user": base.append("deprecated_by IS NULL") if node is None: @@ -120,6 +120,52 @@ def compile_filters( return " AND ".join([*base, compiled]) +def compile_filters_for_decision( + node: FilterNode | None, + *, + owner_id: str, + owner_type: str, + app_id: str = "default", + project_id: str = "default", +) -> str: + """Compile filters for the ``decision`` table. + + Drops ``sender_id`` predicates because the decision schema has no + ``sender_ids`` column (tags are labels, not conversation + participants). Mixed user search compiles two ``where`` strings so + an episode ``sender_id`` filter still applies to the episode lane + without breaking the decision query. Other allowed fields + (``session_id`` / ``timestamp`` / ``parent_id`` / ``parent_type``) + and the user ``deprecated_by IS NULL`` pin are kept. + """ + stripped: FilterNode | None = None + if node is not None: + dumped = _drop_sender_id(node.model_dump(exclude_none=True)) + if dumped: + stripped = FilterNode.model_validate(dumped) + return compile_filters( + stripped, + owner_id=owner_id, + owner_type=owner_type, + app_id=app_id, + project_id=project_id, + ) + + +def _drop_sender_id(raw: Any) -> Any: + """Recursively strip ``sender_id`` keys from a dumped FilterNode dict.""" + if isinstance(raw, list): + return [_drop_sender_id(item) for item in raw] + if not isinstance(raw, dict): + return raw + out: dict[str, Any] = {} + for key, value in raw.items(): + if key == "sender_id": + continue + out[key] = _drop_sender_id(value) + return out + + # ── Internals ──────────────────────────────────────────────────────────── diff --git a/src/everos/memory/search/manager.py b/src/everos/memory/search/manager.py index 1ae5bfd91..c92f1b4f0 100644 --- a/src/everos/memory/search/manager.py +++ b/src/everos/memory/search/manager.py @@ -2,20 +2,22 @@ Hard partition by ``owner_type``: -* ``user`` → ``episodes`` (+ ``profiles`` when ``include_profile=true``) +* ``user`` → ``episodes`` + ``decisions`` (``kinds`` can restrict + either lane; ``profiles`` when ``include_profile=true``; + ``principles`` when ``include_principles=true``) * ``agent`` → ``agent_cases`` + ``agent_skills`` Per kind, :func:`memory.search.adapter.resolve_pipeline` decides whether the path is "single-route recall, no fusion" (``KEYWORD`` / ``VECTOR``) -or "sparse + dense → everalgo.rank" (``HYBRID`` / ``AGENTIC``). Component -guards (embedding / cross-encoder / LLM) raise early when a method is -selected without its prerequisites. +or "sparse + dense → fusion" (``HYBRID``). Episode HYBRID uses the +heap-expand pipeline; Decision HYBRID uses :func:`everalgo.rank.fusion.rrf` +and never ``arank``. ``AGENTIC`` still only drives the episode (and +agent) graphs; the decision lane runs the same sparse+dense+rrf path +so ``kinds=["decision"]`` + agentic is not an empty response. -``HYBRID`` defaults to **no LLM rerank** — the response comes back -straight after the heap-expand pipeline (RRF-ordered expansion → LR-calibrated -global top-N competition with fact eviction). ``enable_llm_rerank`` is -**ignored** for the hierarchy path. ``AGENTIC`` keeps its own -internal cross-encoder rerank loop; the flag is ignored there. +``enable_llm_rerank`` is **ignored** for the episode hierarchy path +and for the decision rrf path. ``AGENTIC`` keeps its own internal +cross-encoder rerank loop; the flag is ignored there. ``SearchEpisodeItem.atomic_facts`` is populated **only** when the HYBRID pipeline runs over episodes. The other methods leave it empty: there is @@ -63,19 +65,22 @@ SearchAgentCaseItem, SearchAgentSkillItem, SearchData, + SearchDecisionItem, SearchEpisodeItem, SearchMethod, + SearchPrincipleItem, SearchProfileItem, SearchRequest, SearchResponse, UnprocessedMessageDTO, ) -from .filters import compile_filters +from .filters import compile_filters, compile_filters_for_decision from .hierarchy import build_ep_to_fact_parents, heap_expand from .shaper import ( reshape_hybrid_output, shape_agent_case_from_candidate, shape_agent_skill_from_candidate, + shape_decision_from_candidate, shape_episode_from_candidate, ) from .skill_hybrid import search_agent_skills_hybrid @@ -91,7 +96,9 @@ AgentCaseRecaller, AgentSkillRecaller, AtomicFactRecaller, + DecisionRecaller, EpisodeRecaller, + PrincipleRecaller, ProfileRecaller, ) @@ -137,9 +144,10 @@ def _top_score(data: SearchData) -> float: """Max relevance score across scored result items (0.0 when empty). - Profiles are excluded — they are a KV fetch with no query-relevance score. + Profiles and principles are excluded — they are KV fetches with no + query-relevance score. """ - items = [*data.episodes, *data.agent_cases, *data.agent_skills] + items = [*data.episodes, *data.decisions, *data.agent_cases, *data.agent_skills] return max((item.score for item in items), default=0.0) @@ -160,9 +168,11 @@ def __init__( *, episode_recaller: EpisodeRecaller, atomic_fact_recaller: AtomicFactRecaller, + decision_recaller: DecisionRecaller, agent_case_recaller: AgentCaseRecaller, agent_skill_recaller: AgentSkillRecaller, profile_recaller: ProfileRecaller, + principle_recaller: PrincipleRecaller, embedding: EmbeddingProvider | None, reranker: RerankProvider | None, llm_client: LLMClient | None, @@ -170,9 +180,11 @@ def __init__( ) -> None: self._ep = episode_recaller self._fact = atomic_fact_recaller + self._decision = decision_recaller self._case = agent_case_recaller self._skill = agent_skill_recaller self._profile = profile_recaller + self._principle = principle_recaller self._embedding = embedding self._reranker = reranker self._llm = llm_client @@ -214,14 +226,31 @@ async def search(self, req: SearchRequest) -> SearchResponse: self._validate_components(req) if req.owner_type == "user": - episodes, profiles, unprocessed = await asyncio.gather( + decision_where = compile_filters_for_decision( + req.filters, + owner_id=req.owner_id, + owner_type=req.owner_type, + app_id=req.app_id, + project_id=req.project_id, + ) + ( + episodes, + decisions, + profiles, + principles, + unprocessed, + ) = await asyncio.gather( self._search_episodes(req, where), + self._search_decisions(req, decision_where), self._fetch_profile(req), + self._fetch_principles(req), self._load_unprocessed(req), ) data = SearchData( episodes=episodes, + decisions=decisions, profiles=profiles, + principles=principles, unprocessed_messages=unprocessed, ) else: # "agent" @@ -241,6 +270,8 @@ async def search(self, req: SearchRequest) -> SearchResponse: span, { "episodes": [e.id for e in data.episodes], + "decisions": [d.id for d in data.decisions], + "principles": [p.id for p in data.principles], "agent_cases": [c.id for c in data.agent_cases], "agent_skills": [s.id for s in data.agent_skills], }, @@ -330,6 +361,8 @@ async def _search_cases_and_skills( async def _search_episodes( self, req: SearchRequest, where: str ) -> list[SearchEpisodeItem]: + if not _wants_episodes(req): + return [] if req.method == SearchMethod.AGENTIC: return await search_episodes_agentic( req.query, @@ -433,6 +466,44 @@ async def _search_episodes( if ep is not None ] + # ── Decisions ─────────────────────────────────────────────────── + + async def _search_decisions( + self, req: SearchRequest, where: str + ) -> list[SearchDecisionItem]: + """User-partition Decision lane. + + ``AGENTIC`` is remapped to HYBRID here: the agentic graph only + serves episodes. Decision still runs sparse + dense + ``rrf`` so + ``kinds=["decision"]`` + agentic is not an empty response. + ``enable_llm_rerank`` is ignored (Decision is not an ``arank`` + ``memory_type``). + """ + if not _wants_decisions(req): + return [] + effective = ( + SearchMethod.HYBRID if req.method == SearchMethod.AGENTIC else req.method + ) + fusion_mode, _ = resolve_pipeline(effective, "decision") + top_k = self._top_k(req.top_k) + + if fusion_mode is None: + cands = await self._single_route_recall(self._decision, req, where, top_k) + shaped = (shape_decision_from_candidate(c) for c in cands[:top_k]) + return [item for item in shaped if item is not None] + + sparse, dense, _ = await self._recall_sparse_dense( + self._decision, req, where, top_k + ) + with memory_span( + "everos.search.rank", + observation_type="span", + metadata={"phase": "rrf", "kind": "decision"}, + ): + fused = rrf(sparse, dense) + shaped = (shape_decision_from_candidate(c) for c in fused[:top_k]) + return [item for item in shaped if item is not None] + # ── Agent cases ───────────────────────────────────────────────── async def _search_agent_cases( @@ -578,11 +649,20 @@ async def _fetch_profile(self, req: SearchRequest) -> list[SearchProfileItem]: return [] return await self._profile.fetch(req.owner_id) + async def _fetch_principles(self, req: SearchRequest) -> list[SearchPrincipleItem]: + if not req.include_principles or req.owner_type != "user": + return [] + return await self._principle.fetch( + req.owner_id, app_id=req.app_id, project_id=req.project_id + ) + # ── Recall helpers ────────────────────────────────────────────── async def _single_route_recall( self, - recaller: EpisodeRecaller | AgentCaseRecaller | AgentSkillRecaller, + recaller: ( + EpisodeRecaller | DecisionRecaller | AgentCaseRecaller | AgentSkillRecaller + ), req: SearchRequest, where: str, top_k: int, @@ -606,7 +686,9 @@ async def _single_route_recall( async def _recall_sparse_dense( self, - recaller: EpisodeRecaller | AgentCaseRecaller | AgentSkillRecaller, + recaller: ( + EpisodeRecaller | DecisionRecaller | AgentCaseRecaller | AgentSkillRecaller + ), req: SearchRequest, where: str, top_k: int, @@ -816,6 +898,12 @@ def _validate_components(self, req: SearchRequest) -> None: """ method = req.method is_agent_hybrid = method == SearchMethod.HYBRID and req.owner_type == "agent" + # AGENTIC's cross-encoder/LLM graph serves episodes (user) and + # cases/skills (agent). kinds=["decision"] remaps to rrf and + # must not demand rerank/LLM. + needs_agentic_graph = method == SearchMethod.AGENTIC and ( + req.owner_type == "agent" or _wants_episodes(req) + ) needs_embedding = method in ( SearchMethod.VECTOR, @@ -849,7 +937,7 @@ def _validate_components(self, req: SearchRequest) -> None: ), ) - if method == SearchMethod.AGENTIC and ( + if needs_agentic_graph and ( not get_rerank_capability().available or self._reranker is None ): raise ProviderNotConfiguredError( @@ -869,7 +957,7 @@ def _validate_components(self, req: SearchRequest) -> None: "rerank lane (needs a configured [rerank] provider)" ), ) - if method == SearchMethod.AGENTIC and self._llm is None: + if needs_agentic_graph and self._llm is None: raise ProviderNotConfiguredError( provider="llm", feature=_feature_name(method, req.owner_type), @@ -890,6 +978,16 @@ def _scored_as_candidate(scored) -> Candidate: # type: ignore[no-untyped-def] ) +def _wants_episodes(req: SearchRequest) -> bool: + """Whether the user-partition episode lane should run.""" + return req.owner_type == "user" and (req.kinds is None or "episode" in req.kinds) + + +def _wants_decisions(req: SearchRequest) -> bool: + """Whether the user-partition decision lane should run.""" + return req.owner_type == "user" and (req.kinds is None or "decision" in req.kinds) + + def _feature_name(method: SearchMethod, owner_type: str) -> str: """Map a search method (+ owner partition) to its 422 ``feature`` tag. diff --git a/src/everos/memory/search/recall/__init__.py b/src/everos/memory/search/recall/__init__.py index 522512e43..3080a6163 100644 --- a/src/everos/memory/search/recall/__init__.py +++ b/src/everos/memory/search/recall/__init__.py @@ -7,10 +7,12 @@ RecallerDeps, EpisodeRecaller, AtomicFactRecaller, + DecisionRecaller, AgentCaseRecaller, AgentSkillRecaller, ProfileRecaller, KnowledgeTopicRecaller, + PrincipleRecaller, ) """ @@ -21,17 +23,21 @@ from .base import RecallerDeps as RecallerDeps from .base import cosine_score_from_distance as cosine_score_from_distance from .base import row_to_candidate as row_to_candidate +from .decision import DecisionRecaller as DecisionRecaller from .episode import EpisodeRecaller as EpisodeRecaller from .knowledge_topic import KnowledgeTopicRecaller as KnowledgeTopicRecaller +from .principle import PrincipleRecaller as PrincipleRecaller from .profile import ProfileRecaller as ProfileRecaller __all__ = [ "AgentCaseRecaller", "AgentSkillRecaller", "AtomicFactRecaller", + "DecisionRecaller", "EpisodeRecaller", "KindRecaller", "KnowledgeTopicRecaller", + "PrincipleRecaller", "ProfileRecaller", "RecallerDeps", "cosine_score_from_distance", diff --git a/src/everos/memory/search/recall/base.py b/src/everos/memory/search/recall/base.py index 9b62c10f9..4abd3bbf9 100644 --- a/src/everos/memory/search/recall/base.py +++ b/src/everos/memory/search/recall/base.py @@ -13,7 +13,7 @@ A shared :class:`RecallerDeps` bundles the providers a recaller needs at construction time (tokenizer for BM25 query, embedder is consumed upstream by the manager so we keep deps minimal). The bundle keeps the -constructor signatures identical across the four LanceDB-backed +constructor signatures identical across the LanceDB-backed recallers so the orchestrator wiring stays uniform. """ @@ -57,10 +57,13 @@ class KindRecaller(Protocol): """One business kind, BM25 + vector recall over its LanceDB table.""" kind: ClassVar[str] - """``episode`` / ``atomic_fact`` / ``agent_case`` / ``agent_skill``.""" + """``episode`` / ``atomic_fact`` / ``decision`` / ``agent_case`` / + ``agent_skill``.""" everalgo_memory_type: ClassVar[str] - """``episodic`` / ``case`` / ``skill`` — passed to ``RankInput.memory_type``.""" + """``episodic`` / ``case`` / ``skill`` — passed to ``RankInput.memory_type``. + Decision leaves this empty: its HYBRID path uses ``rrf``, not ``arank``. + """ text_field: ClassVar[str] """Source column for cross-encoder rerank passages (display text).""" diff --git a/src/everos/memory/search/recall/decision.py b/src/everos/memory/search/recall/decision.py new file mode 100644 index 000000000..326306c54 --- /dev/null +++ b/src/everos/memory/search/recall/decision.py @@ -0,0 +1,120 @@ +"""Decision recaller — dual-column BM25 + cosine ANN. + +The schema declares two BM25 columns (``decision_tokens`` — retrieval +anchor, primary — and ``reason_tokens`` — secondary why-match). +LanceDB's ``nearest_to_text`` searches one column at a time, so we +run the BM25 query twice in parallel and merge by row id keeping the +max score across the two columns. Vector recall is single-shot and +is fed only from the Decision body (cascade embeds that column). + +Mirrors :class:`AgentCaseRecaller` structurally. HYBRID fusion for +this kind is :func:`everalgo.rank.fusion.rrf` in the manager — +``everalgo_memory_type`` is unused because Decision is not an +``arank`` ``memory_type``. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import Sequence +from typing import ClassVar + +from everalgo.types import Candidate + +from everos.infra.persistence.lancedb import Decision, get_table + +from .base import ( + RecallerDeps, + build_or_query_multi_column, + cosine_score_from_distance, + row_to_candidate, +) + + +class DecisionRecaller: + """BM25 (dual-column) + vector recall over the LanceDB ``decision`` table.""" + + kind: ClassVar[str] = "decision" + everalgo_memory_type: ClassVar[str] = "" + """Unused. Decision HYBRID fuses via ``rrf`` and never builds + :class:`~everalgo.types.RankInput`.""" + text_field: ClassVar[str] = "decision" + + def __init__(self, deps: RecallerDeps) -> None: + self._deps = deps + + async def sparse_recall( + self, query: str, where: str, *, limit: int + ) -> list[Candidate]: + """Dual-column BM25 recall via OR-mode BooleanQuery per column. + + Each tokenised term becomes a ``SHOULD`` clause so a single + IDF≈0 token doesn't poison the column query (see + ``EpisodeRecaller.sparse_recall``). One BooleanQuery is built + per BM25 column (``MatchQuery`` is column-bound), then the + two per-column result lists merge by id keeping the max score. + """ + column_queries = build_or_query_multi_column( + self._deps.tokenizer, query, Decision.BM25_FIELDS + ) + if column_queries is None: + return [] + table = await get_table(Decision.TABLE_NAME, Decision) + + async def _query_one(column: str) -> list[dict]: + return ( + await table.query() + .nearest_to_text(column_queries[column]) + .where(where) + .limit(limit) + .to_list() + ) + + per_column = await asyncio.gather( + *(_query_one(col) for col in Decision.BM25_FIELDS), + ) + # Merge by id, keep the max BM25 score across the two columns. + # decision-body hits typically score higher (the retrieval + # anchor); reason hits catch "why did we pick X" queries. + best: dict[str, dict] = {} + for rows in per_column: + for r in rows: + rid = r.get("id") + if not isinstance(rid, str): + continue + score = float(r.get("_score", 0.0)) + existing = best.get(rid) + if existing is None or score > float(existing.get("_score", 0.0)): + merged = dict(r) + merged["_score"] = score + best[rid] = merged + merged_rows = sorted( + best.values(), key=lambda r: float(r.get("_score", 0.0)), reverse=True + )[:limit] + return [ + row_to_candidate(r, source="keyword", score=float(r.get("_score", 0.0))) + for r in merged_rows + ] + + async def dense_recall( + self, vector: Sequence[float], where: str, *, limit: int + ) -> list[Candidate]: + if not vector: + return [] + table = await get_table(Decision.TABLE_NAME, Decision) + rows = ( + await table.query() + .nearest_to(list(vector)) + .distance_type("cosine") + .where(where) + .limit(limit) + .to_list() + ) + return [ + row_to_candidate( + r, + source="vector", + score=cosine_score_from_distance(r.get("_distance")), + ) + for r in rows + ] diff --git a/src/everos/memory/search/recall/principle.py b/src/everos/memory/search/recall/principle.py new file mode 100644 index 000000000..a611924ec --- /dev/null +++ b/src/everos/memory/search/recall/principle.py @@ -0,0 +1,77 @@ +"""Principle recall — KV-by-owner LanceDB fetch (no ranking). + +Principle is Meta Memory, not a product Kind: there is no HYBRID / +BM25 / vector lane and ``kinds`` never accepts ``"principle"``. The +recaller is a deliberate KV lookup: given ``owner_id`` + app/project +scope, return every ``principle`` row for that user (one file exploded +to N rows). There is no ``query`` and no ``score``. + +The cascade keeps ``Principle`` rows in sync with +``users//principles.md``; this recaller just reads them. +Unlike :class:`ProfileRecaller` (PK ``id = owner_id``, at most one +row), principle PKs are ``_`` so the fetch +filters ``owner_id`` **and** ``app_id`` **and** ``project_id``. +""" + +from __future__ import annotations + +from everos.component.utils.datetime import from_timestamp, to_display_tz +from everos.core.observability.logging import get_logger +from everos.infra.persistence.lancedb import principle_repo + +from ..dto import SearchPrincipleItem + +logger = get_logger(__name__) + +_FETCH_LIMIT = 1000 + + +class PrincipleRecaller: + """Fetch the owner's principle rows from LanceDB (N rows, not 1).""" + + async def fetch( + self, + owner_id: str, + *, + app_id: str = "default", + project_id: str = "default", + ) -> list[SearchPrincipleItem]: + """Return the owner's principle items, or ``[]`` when none exist. + + Empty list (rather than 404) lets the caller emit a normal + response with ``principles=[]`` while the user has no synthesised + principles yet. + """ + if not owner_id: + return [] + where = ( + f"owner_id = '{_q(owner_id)}' AND " + f"app_id = '{_q(app_id)}' AND " + f"project_id = '{_q(project_id)}'" + ) + rows = await principle_repo.find_where(where, limit=_FETCH_LIMIT) + if not rows: + logger.debug("principle_fetch_miss", owner_id=owner_id) + return [] + items: list[SearchPrincipleItem] = [] + for row in rows: + ts = from_timestamp(row.timestamp_ms) + items.append( + SearchPrincipleItem( + id=row.id, + user_id=row.owner_id, + app_id=row.app_id, + project_id=row.project_id, + title=row.title, + statement=row.statement, + source_entry_ids=list(row.source_entry_ids), + timestamp=to_display_tz(ts) or ts, + score=None, + ) + ) + return items + + +def _q(value: str) -> str: + """Escape single quotes for a LanceDB SQL-like ``where`` predicate.""" + return value.replace("'", "''") diff --git a/src/everos/memory/search/shaper.py b/src/everos/memory/search/shaper.py index 14bfce6e6..b8c29f853 100644 --- a/src/everos/memory/search/shaper.py +++ b/src/everos/memory/search/shaper.py @@ -3,6 +3,7 @@ This module handles two distinct flows: * Simple kinds — :func:`shape_episode_from_candidate`, + :func:`shape_decision_from_candidate`, :func:`shape_agent_case_from_candidate`, :func:`shape_agent_skill_from_candidate`. Each consumes one :class:`Candidate` and emits the matching ``SearchXxxItem``. @@ -32,6 +33,7 @@ SearchAgentCaseItem, SearchAgentSkillItem, SearchAtomicFactItem, + SearchDecisionItem, SearchEpisodeItem, ) @@ -86,6 +88,55 @@ def shape_episode_from_candidate( ) +# ── Decision shaping ──────────────────────────────────────────────────── + + +def shape_decision_from_candidate(candidate: Candidate) -> SearchDecisionItem | None: + """Build a :class:`SearchDecisionItem` from a recall ``Candidate``. + + Returns ``None`` if the row is malformed (owner_type is not + ``"user"``, or title / decision / reason / timestamp missing). + """ + md = candidate.metadata + if md.get("owner_type") != "user": + logger.warning( + "shape_decision_unexpected_owner_type", + id=candidate.id, + owner_type=md.get("owner_type"), + ) + return None + timestamp = _coerce_datetime(md.get("timestamp")) + if timestamp is None: + logger.warning("shape_decision_missing_timestamp", id=candidate.id) + return None + owner_id = md.get("owner_id") + title = md.get("title") + decision = md.get("decision") + reason = md.get("reason") + if not ( + isinstance(owner_id, str) + and isinstance(title, str) + and isinstance(decision, str) + and isinstance(reason, str) + ): + logger.warning("shape_decision_missing_required_field", id=candidate.id) + return None + return SearchDecisionItem( + id=candidate.id, + user_id=owner_id, + app_id=_as_str(md.get("app_id")) or "default", + project_id=_as_str(md.get("project_id")) or "default", + session_id=_as_optional_str(md.get("session_id")), + timestamp=timestamp, + title=title, + decision=decision, + reason=reason, + impact=_as_optional_str(md.get("impact")), + tags=_as_str_list(md.get("tags")), + score=float(candidate.score), + ) + + # ── Atomic fact shaping ───────────────────────────────────────────────── diff --git a/src/everos/memory/strategies/__init__.py b/src/everos/memory/strategies/__init__.py index 1509bdb06..53a622a83 100644 --- a/src/everos/memory/strategies/__init__.py +++ b/src/everos/memory/strategies/__init__.py @@ -5,9 +5,13 @@ extract_agent_case, extract_agent_skill, extract_atomic_facts, + extract_decision, extract_foresight, + extract_principles, extract_user_profile, + reflect_decisions, reflect_episodes, + trigger_decision_clustering, trigger_profile_clustering, trigger_skill_clustering, ) @@ -16,9 +20,15 @@ from .extract_agent_case import extract_agent_case as extract_agent_case from .extract_agent_skill import extract_agent_skill as extract_agent_skill from .extract_atomic_facts import extract_atomic_facts as extract_atomic_facts +from .extract_decision import extract_decision as extract_decision from .extract_foresight import extract_foresight as extract_foresight +from .extract_principles import extract_principles as extract_principles from .extract_user_profile import extract_user_profile as extract_user_profile +from .reflect_decisions import reflect_decisions as reflect_decisions from .reflect_episodes import reflect_episodes as reflect_episodes +from .trigger_decision_clustering import ( + trigger_decision_clustering as trigger_decision_clustering, +) from .trigger_profile_clustering import ( trigger_profile_clustering as trigger_profile_clustering, ) @@ -30,9 +40,13 @@ "extract_agent_case", "extract_agent_skill", "extract_atomic_facts", + "extract_decision", "extract_foresight", + "extract_principles", "extract_user_profile", + "reflect_decisions", "reflect_episodes", + "trigger_decision_clustering", "trigger_profile_clustering", "trigger_skill_clustering", ] diff --git a/src/everos/memory/strategies/extract_decision.py b/src/everos/memory/strategies/extract_decision.py new file mode 100644 index 000000000..73bc7da29 --- /dev/null +++ b/src/everos/memory/strategies/extract_decision.py @@ -0,0 +1,197 @@ +"""extract_decision strategy — derive Decisions from a fresh MemCell. + +One LLM call per memcell (``DecisionExtractor`` has no ``sender_id``; +every algo ``owner_id`` is ``None``). EverOS then fans the same body +out to every user sender via ``Decision.from_algo``, writes one batched +``append_entries`` per owner, and emits ``DecisionExtracted`` per +written entry so clustering can consume the event bus instead of +racing LanceDB. + +An empty list from the extractor is success: no committed trade-off +in the slice, so no md write and no emit. + +**Enabled by default** (``enabled=True``). Unlike foresight, Search +will consume decisions; running the extractor on every +``UserPipelineStarted`` is the product path. Disable per install in +``ome.toml`` only for evaluation runs that must not spend the extra +tokens: + +.. code-block:: toml + + [strategies.extract_decision] + enabled = false + +The sender scan filters on ``isinstance(m, ChatMessage)`` rather than +reaching for ``m.role``. Only ``ChatMessage`` carries ``role``: +``ToolCallRequest`` has ``sender_id`` without it, ``ToolCallResult`` +has neither. A pure agent trajectory yields no user senders and +returns without an LLM call. +""" + +from __future__ import annotations + +from pathlib import Path + +from everalgo.types import ChatMessage +from everalgo.user_memory import DecisionExtractor + +from everos.component.llm import get_llm_client +from everos.component.utils.datetime import from_timestamp, to_iso_format +from everos.core.observability.logging import get_logger +from everos.core.persistence import MemoryRoot +from everos.infra.ome.context import StrategyContext +from everos.infra.ome.decorator import offline_strategy +from everos.infra.ome.triggers import Immediate +from everos.infra.persistence.markdown import DecisionWriter +from everos.memory.events import DecisionExtracted, UserPipelineStarted +from everos.memory.models import Decision, MemCell +from everos.memory.prompt_slots import PromptLoader + +logger = get_logger(__name__) + +_writer: DecisionWriter | None = None +_prompt_loader: PromptLoader | None = None + + +def _config_root() -> Path: + """Return ``src/everos/config`` (bundled prompt slots).""" + return Path(__file__).resolve().parents[2] / "config" + + +def _get_writer() -> DecisionWriter: + global _writer + if _writer is None: + _writer = DecisionWriter(root=MemoryRoot.resolve()) + return _writer + + +def _get_prompt_loader() -> PromptLoader: + global _prompt_loader + if _prompt_loader is None: + _prompt_loader = PromptLoader(_config_root()) + return _prompt_loader + + +def _unique_user_senders(memcell: MemCell) -> list[str]: + """Distinct role=user sender_ids, preserving first-seen order. + + Skips non-``ChatMessage`` items (agent trajectories' tool calls + have no ``role``). Does not sort — order matches Episode + pipeline ``_unique_user_senders`` so two runs over the same + memcell fan out identically. + """ + senders: list[str] = [] + for item in memcell.items: + if not isinstance(item, ChatMessage) or item.role != "user": + continue + sid = item.sender_id + if sid and sid not in senders: + senders.append(sid) + return senders + + +@offline_strategy( + name="extract_decision", + trigger=Immediate(on=[UserPipelineStarted]), + emits=[DecisionExtracted], + max_retries=2, + enabled=True, +) +async def extract_decision(event: UserPipelineStarted, ctx: StrategyContext) -> None: + owner_ids = _unique_user_senders(event.memcell) + if not owner_ids: + logger.info( + "decisions_extracted", + memcell_id=event.memcell_id, + session_id=event.session_id, + count=0, + owner_ids=[], + ) + return + + prompt = _get_prompt_loader().load("decision_extract") + extractor = DecisionExtractor(llm=get_llm_client()) + algo_decisions = await extractor.aextract(event.memcell, prompt=prompt) + if not algo_decisions: + logger.info( + "decisions_extracted", + memcell_id=event.memcell_id, + session_id=event.session_id, + count=0, + owner_ids=owner_ids, + ) + return + + writer = _get_writer() + written = 0 + for owner_id in owner_ids: + decisions = [ + Decision.from_algo( + algo, + owner_id=owner_id, + session_id=event.session_id, + parent_id=event.memcell_id, + ) + for algo in algo_decisions + ] + items = [_decision_to_entry_body(d) for d in decisions] + eids = await writer.append_entries( + owner_id, + items, + app_id=event.app_id, + project_id=event.project_id, + ) + for d, eid in zip(decisions, eids, strict=True): + await ctx.emit( + DecisionExtracted( + memcell_id=event.memcell_id, + decision_entry_id=eid.format(), + title=d.title, + decision_text=d.decision, + reason=d.reason, + impact=d.impact, + tags=list(d.tags), + decision_timestamp_ms=d.timestamp, + owner_id=d.owner_id, + session_id=event.session_id, + app_id=event.app_id, + project_id=event.project_id, + source="pipeline", + ) + ) + written += 1 + + logger.info( + "decisions_extracted", + memcell_id=event.memcell_id, + session_id=event.session_id, + count=written, + owner_ids=owner_ids, + ) + + +def _decision_to_entry_body( + d: Decision, +) -> tuple[dict[str, object], dict[str, str]]: + """Split a domain Decision into ``(inline, sections)`` for md rendering. + + Lives in the strategy (memory) layer rather than the writer (infra + must not import ``memory``). ``Impact`` is omitted when empty so md + stays compact; ``tags`` is always written so cascade can parse a list. + """ + inline: dict[str, object] = { + "owner_id": d.owner_id, + "session_id": d.session_id, + "timestamp": to_iso_format(from_timestamp(d.timestamp)), + "parent_type": "memcell", + "parent_id": d.parent_id, + "tags": list(d.tags), + } + sections: dict[str, str] = { + "Title": d.title, + "Decision": d.decision, + "Reason": d.reason, + } + if d.impact: + sections["Impact"] = d.impact + return inline, sections diff --git a/src/everos/memory/strategies/extract_principles.py b/src/everos/memory/strategies/extract_principles.py new file mode 100644 index 000000000..9d89d9fe2 --- /dev/null +++ b/src/everos/memory/strategies/extract_principles.py @@ -0,0 +1,309 @@ +"""extract_principles strategy — synthesise principles.md from decision clusters. + +Fires on :class:`DecisionClusterUpdated` (emitted by +``trigger_decision_clustering`` after a sqlite ``kind=decision`` merge). +There is **no** ``DecisionExtracted`` fallback: without embedding the +cluster strategy never emits, so this strategy never runs. That is +intentional — principles are cluster-level Meta Memory. + +``principles.md`` is one file per user. Every dispatch **unions all** +``kind=decision`` clusters for the owner: extracting only the triggering +cluster and rewriting the file would wipe every other cluster's +principles. One LLM call per cluster with loadable members +(:meth:`PrincipleExtractor.aextract`); one ``ProfileWriter.write`` of +the concatenated result. Persist happens only after every cluster +extract succeeds so OME retry is whole-strategy. + +Input shape is ``list[tuple[entry_id, Decision]]`` per cluster. The +triggering member is built from the event snapshot (title / body / +reason / impact / tags / timestamp) so we do not race cascade. Other +members load from markdown SoT via :class:`DecisionReader`, with a +Lance ``decision`` row as a last-resort fallback. Members that cannot +be loaded are skipped (debug log), not a whole-run failure. + +Empty extractor output for a cluster is success. Ids (``pr_<12hex>``) +are minted at write time — EverAlgo ``Principle`` has no id. +""" + +from __future__ import annotations + +from everalgo.clustering import Cluster as AlgoCluster +from everalgo.types import Decision as AlgoDecision +from everalgo.types import Principle as AlgoPrinciple +from everalgo.user_memory import PrincipleExtractor + +from everos.component.llm import get_llm_client +from everos.component.utils.datetime import from_iso_format, to_timestamp_ms +from everos.core.observability.logging import get_logger +from everos.core.persistence import MemoryRoot +from everos.core.persistence.markdown import StructuredEntry +from everos.infra.ome.context import StrategyContext +from everos.infra.ome.decorator import offline_strategy +from everos.infra.ome.triggers import Immediate +from everos.infra.persistence.lancedb import decision_repo +from everos.infra.persistence.markdown import ( + DecisionReader, + PrincipleFrontmatter, + PrincipleItem, + ProfileWriter, + mint_principle_id, + render_principles_body, +) +from everos.infra.persistence.sqlite import cluster_repo +from everos.memory._partition_locks import get_partition_lock +from everos.memory.events import DecisionClusterUpdated + +logger = get_logger(__name__) + +_writer: ProfileWriter | None = None +_decision_reader: DecisionReader | None = None + + +def _get_writer() -> ProfileWriter: + global _writer + if _writer is None: + _writer = ProfileWriter(root=MemoryRoot.resolve()) + return _writer + + +def _get_decision_reader() -> DecisionReader: + global _decision_reader + if _decision_reader is None: + _decision_reader = DecisionReader(root=MemoryRoot.resolve()) + return _decision_reader + + +@offline_strategy( + name="extract_principles", + trigger=Immediate(on=[DecisionClusterUpdated]), + emits=[], + max_retries=2, +) +async def extract_principles( + event: DecisionClusterUpdated, ctx: StrategyContext +) -> None: + # Serialise on owner: principles.md is a single per-user file and the + # body is read-all-clusters → LLM → overwrite. Different users run + # fully in parallel. + partition = f"{event.app_id}:{event.project_id}:{event.owner_id}" + async with get_partition_lock("extract_principles", partition): + clusters = await cluster_repo.list_for_owner( + event.owner_id, + "decision", + app_id=event.app_id, + project_id=event.project_id, + ) + + cluster_pairs: list[list[tuple[str, AlgoDecision]]] = [] + for cluster in clusters: + pairs = await _load_pairs(cluster, event) + if pairs: + cluster_pairs.append(pairs) + + if not cluster_pairs: + if not clusters or not any(c.members for c in clusters): + await _persist_principles( + [], + owner_id=event.owner_id, + app_id=event.app_id, + project_id=event.project_id, + ) + logger.info( + "principles_extracted", + owner_id=event.owner_id, + cluster_count=len(clusters), + principle_count=0, + ) + else: + logger.debug( + "extract_principles_no_loadable_decisions", + owner_id=event.owner_id, + cluster_count=len(clusters), + ) + return + + extractor = PrincipleExtractor(llm=get_llm_client()) + collected: list[AlgoPrinciple] = [] + for pairs in cluster_pairs: + collected.extend(await extractor.aextract(pairs, owner_id=event.owner_id)) + + await _persist_principles( + collected, + owner_id=event.owner_id, + app_id=event.app_id, + project_id=event.project_id, + ) + logger.info( + "principles_extracted", + owner_id=event.owner_id, + cluster_count=len(cluster_pairs), + principle_count=len(collected), + ) + + +async def _load_pairs( + cluster: AlgoCluster, event: DecisionClusterUpdated +) -> list[tuple[str, AlgoDecision]]: + """Resolve ``(entry_id, Decision)`` for one cluster; skip unloadable members.""" + pairs: list[tuple[str, AlgoDecision]] = [] + seen: set[str] = set() + for entry_id in cluster.members: + eid = entry_id.strip() + if not eid or eid in seen: + continue + seen.add(eid) + decision = await _resolve_decision(eid, event) + if decision is None: + logger.debug( + "extract_principles_member_unresolved", + owner_id=event.owner_id, + entry_id=eid, + cluster_id=cluster.id, + ) + continue + pairs.append((eid, decision)) + return pairs + + +async def _resolve_decision( + entry_id: str, event: DecisionClusterUpdated +) -> AlgoDecision | None: + """Snapshot for the triggering row, then md SoT, then Lance fallback.""" + if entry_id == event.decision_entry_id and event.decision_text.strip(): + return _from_snapshot(event) + + structured = await _try_structured(entry_id, event) + if structured is not None: + return structured + + return await _try_lance(entry_id, event) + + +def _from_snapshot(event: DecisionClusterUpdated) -> AlgoDecision: + return AlgoDecision( + owner_id=event.owner_id, + title=event.title, + decision=event.decision_text, + reason=event.reason, + impact=event.impact, + tags=list(event.tags), + timestamp=event.decision_timestamp_ms, + ) + + +async def _try_structured( + entry_id: str, event: DecisionClusterUpdated +) -> AlgoDecision | None: + try: + structured = await _get_decision_reader().find_structured( + event.owner_id, + entry_id, + app_id=event.app_id, + project_id=event.project_id, + ) + except ValueError: + logger.debug( + "extract_principles_entry_id_unparseable", + owner_id=event.owner_id, + entry_id=entry_id, + ) + return None + if structured is None: + return None + return _from_structured(structured, owner_id=event.owner_id) + + +def _from_structured( + structured: StructuredEntry, *, owner_id: str +) -> AlgoDecision | None: + decision = (structured.sections.get("Decision") or "").strip() + if not decision: + return None + impact = (structured.sections.get("Impact") or "").strip() or None + return AlgoDecision( + owner_id=owner_id, + title=(structured.sections.get("Title") or "").strip(), + decision=decision, + reason=(structured.sections.get("Reason") or "").strip(), + impact=impact, + tags=_parse_tags(structured.inline.get("tags") or ""), + timestamp=_timestamp_ms_from_inline(structured.inline.get("timestamp") or ""), + ) + + +async def _try_lance( + entry_id: str, event: DecisionClusterUpdated +) -> AlgoDecision | None: + row = await decision_repo.find_by_owner_entry( + event.owner_id, + entry_id, + app_id=event.app_id, + project_id=event.project_id, + ) + if row is None or not row.decision.strip(): + return None + return AlgoDecision( + owner_id=row.owner_id, + title=row.title, + decision=row.decision, + reason=row.reason, + impact=row.impact, + tags=list(row.tags), + timestamp=to_timestamp_ms(row.timestamp), + ) + + +def _parse_tags(raw: str) -> list[str]: + """Parse ``"[a, b, c]"`` back into ``["a", "b", "c"]``. + + Mirrors cascade ``parse_inline_list`` without importing the handler + package from a strategy. + """ + text = raw.strip() + if not (text.startswith("[") and text.endswith("]")): + return [] + body = text[1:-1].strip() + if not body: + return [] + return [tok.strip() for tok in body.split(",") if tok.strip()] + + +def _timestamp_ms_from_inline(raw: str) -> int: + if not raw.strip(): + return 0 + try: + return to_timestamp_ms(from_iso_format(raw.strip())) + except (TypeError, ValueError): + return 0 + + +async def _persist_principles( + principles: list[AlgoPrinciple], + *, + owner_id: str, + app_id: str, + project_id: str, +) -> None: + """Write the union of extracted principles to ``principles.md``.""" + items = [ + PrincipleItem( + id=mint_principle_id(), + title=p.title, + statement=p.statement, + source_entry_ids=list(p.source_entry_ids), + timestamp_ms=p.timestamp, + ) + for p in principles + ] + frontmatter = PrincipleFrontmatter( + id=f"principle_{owner_id}", + user_id=owner_id, + principles=items, + ) + await _get_writer().write( + owner_id, + frontmatter=frontmatter, + body=render_principles_body(items), + app_id=app_id, + project_id=project_id, + ) diff --git a/src/everos/memory/strategies/reflect_decisions.py b/src/everos/memory/strategies/reflect_decisions.py new file mode 100644 index 000000000..98e7c8ce9 --- /dev/null +++ b/src/everos/memory/strategies/reflect_decisions.py @@ -0,0 +1,97 @@ +"""reflect_decisions Cron strategy — weekly Decision consolidation. + +Triggered by a cron schedule (default: ``0 2 * * 1``). Enumerates all +distinct owner scopes from the cluster table and runs the +:class:`DecisionReflectionOrchestrator` for each. Disabled by default +(same cadence as ``reflect_episodes``). + +The strategy is a thin entry point: it constructs the orchestrator with +production singletons and iterates over owners. All business logic +lives in :mod:`everos.memory.reflection.decision_orchestrator`. +""" + +from __future__ import annotations + +import asyncio + +from everos.component.embedding import get_embedding_capability +from everos.component.llm import get_llm_client +from everos.core.observability.logging import get_logger +from everos.core.persistence import MemoryRoot +from everos.infra.ome.context import StrategyContext +from everos.infra.ome.decorator import offline_strategy +from everos.infra.ome.events import CronTick +from everos.infra.ome.triggers import Cron +from everos.infra.persistence.lancedb import decision_repo +from everos.infra.persistence.markdown import DecisionWriter +from everos.infra.persistence.sqlite import ( + cluster_repo, + reflection_report_repo, +) +from everos.memory.events import DecisionExtracted +from everos.memory.reflection import DecisionReflectionOrchestrator + +logger = get_logger(__name__) + +_writer: DecisionWriter | None = None + + +def _get_writer() -> DecisionWriter: + """Return the lazily-initialised DecisionWriter singleton.""" + global _writer + if _writer is None: + _writer = DecisionWriter(root=MemoryRoot.resolve()) + return _writer + + +@offline_strategy( + name="reflect_decisions", + trigger=Cron(expr="0 2 * * 1"), + emits=[DecisionExtracted], + enabled=False, + max_retries=1, +) +async def reflect_decisions(event: CronTick, ctx: StrategyContext) -> None: + """Run Decision Reflection for all owner scopes. + + Args: + event: Cron tick event (unused; triggers the scheduled run). + ctx: OME strategy context for emit and logging. + """ + # Body-guard: capability is checked here for defensive degradation. + # Reflection re-embeds merged decision text, so it cannot run without + # an embedder — silently no-op instead of raising deep inside the + # orchestrator. Tier upgrades require a server restart; this guard + # is not a hot-reload mechanism. + if not get_embedding_capability().available: + logger.debug( + "strategy_gated_off_embedding_unavailable", + strategy_name="reflect_decisions", + ) + return + + from everalgo.user_memory import DecisionReflector + + orchestrator = DecisionReflectionOrchestrator( + cluster_repo=cluster_repo, + decision_store=decision_repo, + decision_writer=_get_writer(), + report_repo=reflection_report_repo, + reflector=DecisionReflector(llm=get_llm_client()), + embedder=get_embedding_capability().require(), + ) + + owners = await cluster_repo.list_distinct_owners() + await asyncio.gather( + *( + orchestrator.run( + ctx=ctx, + owner_id=owner_id, + owner_type=owner_type, + kind="decision", + app_id=app_id, + project_id=project_id, + ) + for owner_id, owner_type, app_id, project_id in owners + ) + ) diff --git a/src/everos/memory/strategies/trigger_decision_clustering.py b/src/everos/memory/strategies/trigger_decision_clustering.py new file mode 100644 index 000000000..7306e6fd9 --- /dev/null +++ b/src/everos/memory/strategies/trigger_decision_clustering.py @@ -0,0 +1,144 @@ +"""trigger_decision_clustering strategy — group user decisions by topic. + +Listens to :class:`DecisionExtracted` (emitted per written decision after +``extract_decision`` appends its daily-log entry), embeds the +``decision_text``, and merges the resulting size-1 +:class:`everalgo.clustering.Cluster` into the owner's existing decision +cluster set. + +Uses :func:`cluster_by_geometry` (embedding-only cosine + time-window). +Sqlite ``kind`` / ``member_type`` are both ``"decision"`` — not the +user-memory episode track. +""" + +from __future__ import annotations + +import numpy as np +from everalgo.clustering import Cluster as AlgoCluster +from everalgo.clustering import cluster_by_geometry + +from everos.component.embedding import get_embedding_capability +from everos.config import load_settings +from everos.core.observability.logging import get_logger +from everos.infra.ome.context import StrategyContext +from everos.infra.ome.decorator import offline_strategy +from everos.infra.ome.triggers import Immediate +from everos.infra.persistence.sqlite import cluster_repo, mint_cluster_id +from everos.memory._partition_locks import get_partition_lock +from everos.memory.events import DecisionClusterUpdated, DecisionExtracted + +logger = get_logger(__name__) + + +@offline_strategy( + name="trigger_decision_clustering", + trigger=Immediate(on=[DecisionExtracted]), + emits=[DecisionClusterUpdated], + applies_to=lambda e: e.source == "pipeline", + max_retries=2, +) +async def trigger_decision_clustering( + event: DecisionExtracted, ctx: StrategyContext +) -> None: + # Body-guard: capability is checked here for defensive degradation. + # When embedding is unavailable we cannot vectorise the decision, so + # the strategy silently no-ops — no work, no owner lock, no OME + # retry pressure. Same debug-level rationale as + # ``trigger_profile_clustering``: per-dispatch body-guards fire on + # every memorize under Tier 1 and must not flood structured logs. + if not get_embedding_capability().available: + logger.debug( + "strategy_gated_off_embedding_unavailable", + strategy_name="trigger_decision_clustering", + owner_id=event.owner_id, + ) + return + + # Serialise on owner_id: the strategy reads the owner's full cluster + # set, picks merge target by geometry, then upserts — concurrent runs + # on the same owner_id would race the read → decide → write cycle. + # Different users run fully in parallel. + # Lock per (app, project, owner): clusters are scoped to a space, so a + # different space's run must not serialise on (or merge into) this one. + partition = f"{event.app_id}:{event.project_id}:{event.owner_id}" + async with get_partition_lock("trigger_decision_clustering", partition): + # 1. Embed the decision_text into a vector. + # ``.require()`` is defensive: the body-guard above already + # returned when the capability was missing, so this cannot raise + # in the guarded path. Routing through the capability keeps a + # single shared provider (one client, one semaphore) per process. + embedder = get_embedding_capability().require() + vector_list = await embedder.embed(event.decision_text) + vector = np.asarray(vector_list, dtype=np.float32) + + # 2. Load this owner's existing decision clusters (scoped to space). + existing = await cluster_repo.list_for_owner( + event.owner_id, + "decision", + app_id=event.app_id, + project_id=event.project_id, + ) + + # 3. Build a size-1 cluster for the new decision. + new_cluster = AlgoCluster( + id=mint_cluster_id(), + centroid=vector, + count=1, + last_ts=event.decision_timestamp_ms, + preview=[event.decision_text], + members=[event.decision_entry_id], + ) + + # 4. Geometry-merge it into an existing cluster (or keep as-is). + # ``cluster_by_geometry`` is a pure synchronous CPU function (cosine + + # time-window math, no I/O) returning ``Cluster | None`` directly, so + # it must not be awaited (``await None`` raises when there is no + # existing cluster to merge into). + settings = load_settings() + merged = cluster_by_geometry( + new_cluster, + existing, + threshold=settings.clustering.threshold, + time_window_days=settings.clustering.time_window_days, + ) + to_save = merged if merged is not None else new_cluster + + # 5. Persist the (possibly-merged) cluster back to SQLite. + await cluster_repo.upsert_with_members( + to_save, + owner_id=event.owner_id, + owner_type="user", + kind="decision", + member_type="decision", + app_id=event.app_id, + project_id=event.project_id, + ) + + # 6. Emit DecisionClusterUpdated with a row snapshot so a later + # principle extractor can consume the triggering decision without + # racing cascade / polling LanceDB. + assert to_save.id is not None # both branches above set id + await ctx.emit( + DecisionClusterUpdated( + memcell_id=event.memcell_id, + decision_entry_id=event.decision_entry_id, + cluster_id=to_save.id, + owner_id=event.owner_id, + app_id=event.app_id, + project_id=event.project_id, + title=event.title, + decision_text=event.decision_text, + reason=event.reason, + impact=event.impact, + tags=list(event.tags), + decision_timestamp_ms=event.decision_timestamp_ms, + ) + ) + logger.info( + "decision_cluster_updated", + memcell_id=event.memcell_id, + cluster_id=to_save.id, + owner_id=event.owner_id, + merged=merged is not None, + cluster_count=to_save.count, + ) diff --git a/src/everos/service/get.py b/src/everos/service/get.py index 6d5a72bc2..105da676e 100644 --- a/src/everos/service/get.py +++ b/src/everos/service/get.py @@ -14,6 +14,7 @@ from everos.infra.persistence.lancedb import ( agent_case_repo, agent_skill_repo, + decision_repo, episode_repo, user_profile_repo, ) @@ -29,6 +30,7 @@ def _get_manager() -> GetManager: if _manager is None: _manager = GetManager( episode_repo=episode_repo, + decision_repo=decision_repo, agent_case_repo=agent_case_repo, agent_skill_repo=agent_skill_repo, user_profile_repo=user_profile_repo, diff --git a/src/everos/service/memorize.py b/src/everos/service/memorize.py index 2820be153..667603c45 100644 --- a/src/everos/service/memorize.py +++ b/src/everos/service/memorize.py @@ -46,9 +46,13 @@ extract_agent_case, extract_agent_skill, extract_atomic_facts, + extract_decision, extract_foresight, + extract_principles, extract_user_profile, + reflect_decisions, reflect_episodes, + trigger_decision_clustering, trigger_profile_clustering, trigger_skill_clustering, ) @@ -114,9 +118,11 @@ def _get_agent_pipeline() -> AgentMemoryPipeline: _STRATEGIES_ALWAYS = ( extract_atomic_facts, + extract_decision, extract_foresight, extract_agent_case, extract_user_profile, + extract_principles, ) """LLM-only strategies — no embed dependency, always registered.""" @@ -124,7 +130,9 @@ def _get_agent_pipeline() -> AgentMemoryPipeline: trigger_skill_clustering, extract_agent_skill, trigger_profile_clustering, + trigger_decision_clustering, reflect_episodes, + reflect_decisions, ) """Strategies whose body re-embeds or consumes a re-embedded cluster. diff --git a/src/everos/service/search.py b/src/everos/service/search.py index 8cb6ee85d..8be83f73f 100644 --- a/src/everos/service/search.py +++ b/src/everos/service/search.py @@ -33,7 +33,9 @@ AgentCaseRecaller, AgentSkillRecaller, AtomicFactRecaller, + DecisionRecaller, EpisodeRecaller, + PrincipleRecaller, ProfileRecaller, RecallerDeps, ) @@ -78,9 +80,11 @@ def _get_manager() -> SearchManager: _manager = SearchManager( episode_recaller=EpisodeRecaller(deps), atomic_fact_recaller=AtomicFactRecaller(deps), + decision_recaller=DecisionRecaller(deps), agent_case_recaller=AgentCaseRecaller(deps), agent_skill_recaller=AgentSkillRecaller(deps), profile_recaller=ProfileRecaller(), + principle_recaller=PrincipleRecaller(), embedding=get_embedding_capability().provider, reranker=get_rerank_capability().provider, llm_client=_get_llm_client(), diff --git a/tests/_consistency_assertions.py b/tests/_consistency_assertions.py index 5890cbaa1..ae51eee22 100644 --- a/tests/_consistency_assertions.py +++ b/tests/_consistency_assertions.py @@ -11,9 +11,10 @@ This is the e2e tail check meant to follow ``add+flush+cascade-drain`` pipelines (see ``tests/e2e/test_add_flush_*_pipeline_e2e.py``). It exercises every kind that writes md and indexes into LanceDB, not just -the 4 daily-log kinds covered by the white-box integration test. +the 5 daily-log kinds covered by the white-box integration test. -Daily-log kinds (atomic_fact / episode / foresight / agent_case) hold +Daily-log kinds (atomic_fact / episode / foresight / decision / +agent_case) hold many entries per md and use a per-entry digest; user_profile + agent_skill are single-md-per-row and digest the file as a whole (agent_skill additionally folds in concatenated ``references/*.md``). diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py index cb88e931a..97f019019 100644 --- a/tests/e2e/conftest.py +++ b/tests/e2e/conftest.py @@ -69,9 +69,12 @@ _STRATEGY_SINGLETONS: tuple[tuple[str, tuple[str, ...]], ...] = ( ("everos.memory.strategies.extract_atomic_facts", ("_writer",)), ("everos.memory.strategies.extract_foresight", ("_writer",)), + ("everos.memory.strategies.extract_decision", ("_writer",)), ("everos.memory.strategies.extract_user_profile", ("_writer", "_reader")), + ("everos.memory.strategies.extract_principles", ("_writer", "_decision_reader")), ("everos.memory.strategies.extract_agent_case", ("_writer",)), ("everos.memory.strategies.extract_agent_skill", ("_writer",)), + ("everos.memory.strategies.reflect_decisions", ("_writer",)), ) diff --git a/tests/e2e/test_get_endpoint_e2e.py b/tests/e2e/test_get_endpoint_e2e.py index 55af0989e..001cf0ca4 100644 --- a/tests/e2e/test_get_endpoint_e2e.py +++ b/tests/e2e/test_get_endpoint_e2e.py @@ -179,6 +179,7 @@ async def test_get_episodes_returns_page_and_total( assert data["episodes"][1]["id"] == "u1_ep_004" # The non-requested kinds are empty arrays (envelope invariant). assert data["profiles"] == [] + assert data["decisions"] == [] assert data["agent_cases"] == [] assert data["agent_skills"] == [] diff --git a/tests/e2e/test_search_endpoint_e2e.py b/tests/e2e/test_search_endpoint_e2e.py index b329b549c..3833cb531 100644 --- a/tests/e2e/test_search_endpoint_e2e.py +++ b/tests/e2e/test_search_endpoint_e2e.py @@ -1220,11 +1220,11 @@ async def test_search_rejects_invalid_request( async def test_search_returns_empty_envelope_for_unknown_owner( client: AsyncClient, ) -> None: - """Owner with no seeded rows → 200 with all five arrays empty. + """Owner with no seeded rows → 200 with all kind arrays empty. - Pins the envelope-shape invariant: ``data.{episodes, profiles, - agent_cases, agent_skills, unprocessed_messages}`` always exist; - an empty result is a successful response, not 404. + Pins the envelope-shape invariant: ``data.{episodes, decisions, + profiles, agent_cases, agent_skills, unprocessed_messages}`` always + exist; an empty result is a successful response, not 404. """ resp = await _post(client, owner_id="ghost", query="anything", method="keyword") assert resp.status_code == 200 @@ -1234,6 +1234,7 @@ async def test_search_returns_empty_envelope_for_unknown_owner( data = body["data"] assert data == { "episodes": [], + "decisions": [], "profiles": [], "agent_cases": [], "agent_skills": [], diff --git a/tests/integration/search/conftest.py b/tests/integration/search/conftest.py index 759599e60..6f95e6169 100644 --- a/tests/integration/search/conftest.py +++ b/tests/integration/search/conftest.py @@ -97,12 +97,14 @@ def _reset_memorize_singletons(mp: pytest.MonkeyPatch) -> None: client_mod = importlib.import_module("everos.component.llm.client") af_mod = importlib.import_module("everos.memory.strategies.extract_atomic_facts") fs_mod = importlib.import_module("everos.memory.strategies.extract_foresight") + dc_mod = importlib.import_module("everos.memory.strategies.extract_decision") for attr in _MEMORIZE_SINGLETONS: mp.setattr(svc, attr, None, raising=False) mp.setattr(client_mod, "_llm_client", None, raising=False) mp.setattr(af_mod, "_writer", None, raising=False) mp.setattr(fs_mod, "_writer", None, raising=False) + mp.setattr(dc_mod, "_writer", None, raising=False) # ── Session corpus: ingest once ──────────────────────────────────────── diff --git a/tests/integration/test_cascade_all_kinds_consistency.py b/tests/integration/test_cascade_all_kinds_consistency.py index c2310ecf3..5599dd479 100644 --- a/tests/integration/test_cascade_all_kinds_consistency.py +++ b/tests/integration/test_cascade_all_kinds_consistency.py @@ -1,4 +1,4 @@ -"""Strict md <-> lancedb consistency across all 4 daily-log kinds. +"""Strict md <-> lancedb consistency across all 5 daily-log kinds. For each registered daily-log kind, seed N entries via the kind's writer, wait for the cascade to drain, then assert exact equality @@ -35,6 +35,7 @@ from everos.infra.persistence.lancedb import ( agent_case_repo, atomic_fact_repo, + decision_repo, dispose_connection, ensure_business_indexes, episode_repo, @@ -43,11 +44,13 @@ from everos.infra.persistence.lancedb.lancedb_manager import get_table from everos.infra.persistence.lancedb.tables.agent_case import AgentCase from everos.infra.persistence.lancedb.tables.atomic_fact import AtomicFact +from everos.infra.persistence.lancedb.tables.decision import Decision from everos.infra.persistence.lancedb.tables.episode import Episode from everos.infra.persistence.lancedb.tables.foresight import Foresight from everos.infra.persistence.markdown import ( AgentCaseWriter, AtomicFactWriter, + DecisionWriter, EpisodeWriter, ForesightWriter, ) @@ -161,6 +164,24 @@ def _fs_item(scope_id: str, j: int): ) +def _dc_item(scope_id: str, j: int): + return ( + { + "owner_id": scope_id, + "session_id": f"s_{j}", + "timestamp": "2026-05-19T07:04:26+00:00", + "parent_id": f"mc_{j}", + "tags": ["runtime", "rust"], + }, + { + "Title": f"title {j}", + "Decision": f"decision body {j}", + "Reason": f"reason {j}", + "Impact": f"impact {j}", + }, + ) + + def _ac_item(scope_id: str, j: int): return ( { @@ -209,6 +230,16 @@ def _ac_item(scope_id: str, j: int): table_cls=Foresight, build_item=_fs_item, ), + _DailyLogKindCase( + name="decision", + scope="users", + dir_name="decisions", + file_prefix="decision", + writer_factory=DecisionWriter, + repo=decision_repo, + table_cls=Decision, + build_item=_dc_item, + ), _DailyLogKindCase( name="agent_case", scope="agents", diff --git a/tests/integration/test_decision_closed_loop.py b/tests/integration/test_decision_closed_loop.py new file mode 100644 index 000000000..6da7d5a56 --- /dev/null +++ b/tests/integration/test_decision_closed_loop.py @@ -0,0 +1,332 @@ +"""Gate 7 — FakeLLM Decision closed loop. + +Input 「我们决定使用 Rust 实现设备 Runtime。」 → extract_decision → +markdown daily-log → cascade Lance ``decision`` → keyword search +「设备 Runtime 为什么使用 Rust？」 recalls ``data.decisions``. + +No real LLM / embedder. ``DecisionExtractor`` is the real class; only +atomic-fact / profile extractors are stubbed so their prompts cannot +steal FakeLLM turns. ``trigger_decision_clustering`` no-ops because +the suite's autouse fixture leaves embedding unavailable. +""" + +from __future__ import annotations + +import asyncio +import importlib +import json +from collections.abc import AsyncIterator +from pathlib import Path +from typing import Any +from unittest.mock import AsyncMock + +import pytest +import pytest_asyncio +from everalgo.llm.types import ChatMessage as LLMChatMessage +from everalgo.llm.types import ChatResponse +from everalgo.testing.fake_llm import FakeLLMClient +from pydantic import ValidationError +from sqlmodel import SQLModel + +from everos.component.tokenizer import build_tokenizer +from everos.core.persistence import MemoryRoot +from everos.infra.persistence.lancedb import ( + decision_repo, + dispose_connection, + ensure_business_indexes, +) +from everos.infra.persistence.sqlite import ( + dispose_engine, + get_engine, + md_change_state_repo, +) +from everos.memory.cascade import CascadeConfig, CascadeOrchestrator +from everos.memory.get import GetMemoryType, GetRequest +from everos.memory.search import SearchMethod, SearchRequest +from everos.service.memorize import memorize + +_OWNER = "u_alice" +_SESSION = "s_decision_loop" +_USER_UTTERANCE = "我们决定使用 Rust 实现设备 Runtime。" +_SEARCH_QUERY = "设备 Runtime 为什么使用 Rust？" + +_DECISION_JSON = json.dumps( + { + "decisions": [ + { + "title": "Device Runtime language", + "decision": "Use Rust for the device Runtime.", + "reason": "Rust gives stable, low-overhead device Runtime.", + "impact": "Device capabilities talk to the Agent Runtime over APIs.", + "tags": ["architecture", "runtime"], + } + ] + } +) + + +def _boundary_json() -> str: + return json.dumps({"reasoning": "test", "boundaries": [], "should_wait": False}) + + +def _episode_json() -> str: + return json.dumps( + { + "title": "Device Runtime language choice", + "content": "The team chose Rust for the device Runtime.", + "summary": "Rust for device Runtime.", + } + ) + + +def _make_fake_llm() -> FakeLLMClient: + """Dispatch by prompt fingerprint so episode / decision never share a queue.""" + + def handler(messages: list[LLMChatMessage], **_: Any) -> ChatResponse: + prompt = messages[0].content if messages else "" + text = prompt.lower() if isinstance(prompt, str) else "" + if "committed decisions" in text: + return ChatResponse(content=_DECISION_JSON, model="fake") + if "boundaries" in text or "should_wait" in text: + return ChatResponse(content=_boundary_json(), model="fake") + return ChatResponse(content=_episode_json(), model="fake") + + return FakeLLMClient(handler=handler) + + +def _decision_md_files(root: Path) -> list[Path]: + base = root / "default_app" / "default_project" / "users" / _OWNER / "decisions" + if not base.is_dir(): + return [] + return sorted(base.glob("decision-*.md")) + + +@pytest_asyncio.fixture +async def closed_loop_env( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> AsyncIterator[Path]: + """Tmp memory root + OME + cascade; FakeLLM; real DecisionExtractor.""" + monkeypatch.setenv("EVEROS_ROOT", str(tmp_path)) + monkeypatch.setenv("EVEROS_MEMORIZE__MODE", "chat") + monkeypatch.setenv("EVEROS_LLM__API_KEY", "fake-key") + monkeypatch.setenv("EVEROS_LLM__BASE_URL", "https://fake.example.com") + monkeypatch.setattr( + MemoryRoot, "resolve", classmethod(lambda cls: MemoryRoot(root=tmp_path)) + ) + (tmp_path / ".index" / "sqlite").mkdir(parents=True, exist_ok=True) + (tmp_path / "ome.toml").write_text("# test\n") + + from everos.config import load_settings + + load_settings.cache_clear() + + from everos.core.persistence.lancedb.repository import LanceRepoBase + + LanceRepoBase._reset_locks_for_tests() + + await dispose_connection() + await dispose_engine() + engine = get_engine() + async with engine.begin() as conn: + await conn.run_sync(SQLModel.metadata.create_all) + await ensure_business_indexes() + + svc = importlib.import_module("everos.service.memorize") + search_svc = importlib.import_module("everos.service.search") + get_svc = importlib.import_module("everos.service.get") + client_mod = importlib.import_module("everos.component.llm.client") + dc_mod = importlib.import_module("everos.memory.strategies.extract_decision") + af_mod = importlib.import_module("everos.memory.strategies.extract_atomic_facts") + prof_mod = importlib.import_module("everos.memory.strategies.extract_user_profile") + + for attr in ( + "_episode_writer", + "_prompt_loader", + "_user_pipeline", + "_agent_pipeline", + "_ome_engine", + ): + monkeypatch.setattr(svc, attr, None, raising=False) + monkeypatch.setattr(search_svc, "_manager", None, raising=False) + monkeypatch.setattr(get_svc, "_manager", None, raising=False) + monkeypatch.setattr(client_mod, "_llm_client", _make_fake_llm(), raising=False) + monkeypatch.setattr(dc_mod, "_writer", None, raising=False) + monkeypatch.setattr(af_mod, "_writer", None, raising=False) + monkeypatch.setattr(prof_mod, "_writer", None, raising=False) + monkeypatch.setattr(prof_mod, "_reader", None, raising=False) + monkeypatch.setattr(prof_mod, "PROFILE_MIN_MEMCELLS", 99, raising=False) + monkeypatch.setattr( + af_mod, + "AtomicFactExtractor", + lambda *a, **k: type( + "M", + (), + {"aextract_from_text": AsyncMock(return_value=[])}, + )(), + ) + + ome = svc._get_engine() + await ome.start() + + cascade = CascadeOrchestrator( + memory_root=MemoryRoot.resolve(), + tokenizer=build_tokenizer(), + config=CascadeConfig( + scan_interval_seconds=0.5, + worker_batch_size=10, + worker_max_retry=1, + worker_poll_interval_seconds=0.05, + worker_retry_backoff_seconds=0.0, + ), + ) + await cascade.start() + await asyncio.sleep(0.3) + + try: + yield tmp_path + finally: + await cascade.stop() + await ome.stop() + await dispose_connection() + await dispose_engine() + monkeypatch.setattr(search_svc, "_manager", None, raising=False) + monkeypatch.setattr(get_svc, "_manager", None, raising=False) + + +async def _poll(condition, *, deadline: float = 20.0) -> Any: # type: ignore[no-untyped-def] + async with asyncio.timeout(deadline): + while True: + result = ( + await condition() + if asyncio.iscoroutinefunction(condition) + else condition() + ) + if result: + return result + await asyncio.sleep(0.05) + + +async def _wait_decision_md(root: Path) -> Path: + def _ready() -> Path | None: + files = _decision_md_files(root) + for path in files: + body = path.read_text(encoding="utf-8") + if "dc_" in body and "Rust" in body and "### Title" in body: + return path + return None + + found = await _poll(_ready) + assert isinstance(found, Path) + return found + + +async def _wait_cascade_done(md_path: str) -> None: + async def _done() -> bool: + row = await md_change_state_repo.get_by_id(md_path) + if row is not None and row.status == "failed": + raise AssertionError(f"cascade failed for {md_path}: {row.error}") + return row is not None and row.status == "done" and row.error is None + + await _poll(_done) + + +async def test_extract_md_cascade_search_recalls_rust_runtime( + closed_loop_env: Path, +) -> None: + """Design §12.2: extract → md → cascade → search recalls the Decision.""" + result = await memorize( + { + "session_id": _SESSION, + "messages": [ + { + "sender_id": _OWNER, + "role": "user", + "content": _USER_UTTERANCE, + "timestamp": 1_700_000_000_000, + }, + { + "sender_id": "assistant", + "role": "assistant", + "content": "Agreed. Rust on the device Runtime.", + "timestamp": 1_700_000_001_000, + }, + ], + }, + is_final=True, + ) + assert result.status == "extracted" + + md_file = await _wait_decision_md(closed_loop_env) + body = md_file.read_text(encoding="utf-8") + assert "dc_" in body + assert "Device Runtime language" in body + assert "Use Rust for the device Runtime." in body + assert "**parent_type**: memcell" in body + assert f"**owner_id**: {_OWNER}" in body + + rel = md_file.relative_to(closed_loop_env).as_posix() + await _wait_cascade_done(rel) + + rows = await decision_repo.find_by_owner(_OWNER) + assert len(rows) >= 1 + hit = next(r for r in rows if "Rust" in r.decision) + assert hit.entry_id.startswith("dc_") + assert hit.deprecated_by is None + assert hit.parent_type == "memcell" + assert "Runtime" in hit.decision + + from everos.service.get import get as get_memory + from everos.service.search import search as search_memory + + async def _keyword_hit(): + resp = await search_memory( + SearchRequest( + user_id=_OWNER, + query=_SEARCH_QUERY, + method=SearchMethod.KEYWORD, + top_k=5, + ) + ) + return resp if resp.data.decisions else None + + search_resp = await _poll(_keyword_hit, deadline=10.0) + assert search_resp.data.episodes is not None + assert search_resp.data.principles == [] + assert search_resp.data.profiles == [] + recalled = search_resp.data.decisions[0] + blob = f"{recalled.title} {recalled.decision} {recalled.reason}" + assert "Rust" in blob + assert "Runtime" in blob + + episode_only = await search_memory( + SearchRequest( + user_id=_OWNER, + query=_SEARCH_QUERY, + method=SearchMethod.KEYWORD, + kinds=["episode"], + top_k=5, + ) + ) + assert episode_only.data.decisions == [] + + get_resp = await get_memory( + GetRequest(user_id=_OWNER, memory_type=GetMemoryType.DECISION) + ) + assert get_resp.data.decisions + listed = get_resp.data.decisions[0] + assert listed.decision == hit.decision + assert listed.title == hit.title + + +def test_kinds_principle_still_rejected() -> None: + with pytest.raises(ValidationError): + SearchRequest( + user_id=_OWNER, + query="x", + kinds=["principle"], # type: ignore[list-item] + ) + + +def test_get_memory_type_principle_still_rejected() -> None: + with pytest.raises(ValidationError): + GetRequest.model_validate({"user_id": _OWNER, "memory_type": "principle"}) diff --git a/tests/integration/test_memorize_agent_mode.py b/tests/integration/test_memorize_agent_mode.py index 23497f806..5e3b3dc82 100644 --- a/tests/integration/test_memorize_agent_mode.py +++ b/tests/integration/test_memorize_agent_mode.py @@ -49,7 +49,8 @@ def handler(messages: list[LLMChatMessage], **_: Any) -> ChatResponse: cuts = queue.pop(0) if queue else [] return ChatResponse(content=_boundary_response(cuts), model="fake") return ChatResponse( - content=json.dumps({"title": "T", "content": "B"}), model="fake" + content=json.dumps({"title": "T", "content": "B", "summary": "B"}), + model="fake", ) return FakeLLMClient(handler=handler) @@ -111,6 +112,7 @@ async def memorize_env( svc = importlib.import_module("everos.service.memorize") af_mod = importlib.import_module("everos.memory.strategies.extract_atomic_facts") fs_mod = importlib.import_module("everos.memory.strategies.extract_foresight") + dc_mod = importlib.import_module("everos.memory.strategies.extract_decision") ac_mod = importlib.import_module("everos.memory.strategies.extract_agent_case") client_mod = importlib.import_module("everos.component.llm.client") @@ -125,6 +127,7 @@ async def memorize_env( monkeypatch.setattr(client_mod, "_llm_client", None, raising=False) monkeypatch.setattr(af_mod, "_writer", None, raising=False) monkeypatch.setattr(fs_mod, "_writer", None, raising=False) + monkeypatch.setattr(dc_mod, "_writer", None, raising=False) started: dict[str, Any] = {"engine": None} @@ -146,10 +149,10 @@ async def _setup(*, mode: str = "agent", fake_llm: FakeLLMClient) -> None: await conn.run_sync(SQLModel.metadata.create_all) started["dispose"] = dispose_engine - # Silence OME strategies so agent_case / atomic / foresight don't - # try real extraction logic during these tests. + # Silence OME strategies so agent_case / atomic / foresight / + # decision don't try real extraction logic during these tests. noop = AsyncMock(return_value=[]) - for mod in (af_mod, fs_mod, ac_mod): + for mod in (af_mod, fs_mod, dc_mod, ac_mod): extractor_attr = next( (n for n in dir(mod) if n.endswith("Extractor")), None ) diff --git a/tests/integration/test_memorize_concurrent_session_lock.py b/tests/integration/test_memorize_concurrent_session_lock.py index b50b1db2b..0ac69e8ab 100644 --- a/tests/integration/test_memorize_concurrent_session_lock.py +++ b/tests/integration/test_memorize_concurrent_session_lock.py @@ -60,7 +60,7 @@ def _boundary_response(boundaries: list[int]) -> str: def _episode_response(title: str = "T", content: str = "B") -> str: - return json.dumps({"title": title, "content": content}) + return json.dumps({"title": title, "content": content, "summary": content}) def _make_extract_all_llm() -> FakeLLMClient: @@ -100,6 +100,7 @@ async def memorize_env_locked( svc = importlib.import_module("everos.service.memorize") af_mod = importlib.import_module("everos.memory.strategies.extract_atomic_facts") fs_mod = importlib.import_module("everos.memory.strategies.extract_foresight") + dc_mod = importlib.import_module("everos.memory.strategies.extract_decision") client_mod = importlib.import_module("everos.component.llm.client") lock_mod = importlib.import_module("everos.service._session_lock") @@ -115,6 +116,7 @@ async def memorize_env_locked( monkeypatch.setattr(client_mod, "_llm_client", None, raising=False) monkeypatch.setattr(af_mod, "_writer", None, raising=False) monkeypatch.setattr(fs_mod, "_writer", None, raising=False) + monkeypatch.setattr(dc_mod, "_writer", None, raising=False) lock_mod._reset_for_tests() started: dict[str, Any] = {"engine": None} @@ -139,6 +141,7 @@ async def _setup(*, fake_llm: FakeLLMClient) -> None: # memcell + buffer cycle; downstream strategies are a separate story). mock_af = AsyncMock(return_value=[]) mock_fs = AsyncMock(return_value=[]) + mock_dc = AsyncMock(return_value=[]) monkeypatch.setattr( af_mod, "AtomicFactExtractor", @@ -149,6 +152,11 @@ async def _setup(*, fake_llm: FakeLLMClient) -> None: "ForesightExtractor", lambda *a, **k: type("M", (), {"aextract": mock_fs})(), ) + monkeypatch.setattr( + dc_mod, + "DecisionExtractor", + lambda *a, **k: type("M", (), {"aextract": mock_dc})(), + ) engine = svc._get_engine() await engine.start() diff --git a/tests/integration/test_memorize_integration.py b/tests/integration/test_memorize_integration.py index bc866e19f..8ea747d79 100644 --- a/tests/integration/test_memorize_integration.py +++ b/tests/integration/test_memorize_integration.py @@ -52,7 +52,7 @@ def _boundary_response(boundaries: list[int]) -> str: def _episode_response(title: str = "Test Subject", content: str = "Test body") -> str: """Build an ``EpisodeExtractor`` JSON response (algo schema).""" - return json.dumps({"title": title, "content": content}) + return json.dumps({"title": title, "content": content, "summary": content}) def _make_fake_llm( @@ -65,7 +65,7 @@ def _make_fake_llm( Pops one ``boundaries=...`` from ``boundary_responses`` per boundary prompt seen; every episode prompt returns the same canned - ``{title, content}``. + ``{title, content, summary}``. """ boundary_queue: list[list[int]] = list(boundary_responses or []) @@ -115,6 +115,7 @@ async def test_x(memorize_env): svc = importlib.import_module("everos.service.memorize") af_mod = importlib.import_module("everos.memory.strategies.extract_atomic_facts") fs_mod = importlib.import_module("everos.memory.strategies.extract_foresight") + dc_mod = importlib.import_module("everos.memory.strategies.extract_decision") client_mod = importlib.import_module("everos.component.llm.client") # Reset singletons. @@ -129,6 +130,7 @@ async def test_x(memorize_env): monkeypatch.setattr(client_mod, "_llm_client", None, raising=False) monkeypatch.setattr(af_mod, "_writer", None, raising=False) monkeypatch.setattr(fs_mod, "_writer", None, raising=False) + monkeypatch.setattr(dc_mod, "_writer", None, raising=False) started: dict[str, Any] = {"engine": None, "sqlite_engine": None} @@ -170,6 +172,7 @@ async def _setup( # (the strategy itself still runs; it just sees no facts/foresights). mock_af = AsyncMock(return_value=[]) mock_fs = AsyncMock(return_value=[]) + mock_dc = AsyncMock(return_value=[]) monkeypatch.setattr( af_mod, "AtomicFactExtractor", @@ -180,6 +183,11 @@ async def _setup( "ForesightExtractor", lambda *a, **k: type("M", (), {"aextract": mock_fs})(), ) + monkeypatch.setattr( + dc_mod, + "DecisionExtractor", + lambda *a, **k: type("M", (), {"aextract": mock_dc})(), + ) engine = svc._get_engine() await engine.start() diff --git a/tests/integration/test_memorize_window_segmentation.py b/tests/integration/test_memorize_window_segmentation.py index 705b68e82..c6f53fca6 100644 --- a/tests/integration/test_memorize_window_segmentation.py +++ b/tests/integration/test_memorize_window_segmentation.py @@ -62,7 +62,7 @@ def _boundary_response(boundaries: list[int]) -> str: def _episode_response(title: str = "T", content: str = "B") -> str: - return json.dumps({"title": title, "content": content}) + return json.dumps({"title": title, "content": content, "summary": content}) def _make_scripted_llm( @@ -103,6 +103,7 @@ async def memorize_env_scripted( svc = importlib.import_module("everos.service.memorize") af_mod = importlib.import_module("everos.memory.strategies.extract_atomic_facts") fs_mod = importlib.import_module("everos.memory.strategies.extract_foresight") + dc_mod = importlib.import_module("everos.memory.strategies.extract_decision") client_mod = importlib.import_module("everos.component.llm.client") lock_mod = importlib.import_module("everos.service._session_lock") @@ -117,6 +118,7 @@ async def memorize_env_scripted( monkeypatch.setattr(client_mod, "_llm_client", None, raising=False) monkeypatch.setattr(af_mod, "_writer", None, raising=False) monkeypatch.setattr(fs_mod, "_writer", None, raising=False) + monkeypatch.setattr(dc_mod, "_writer", None, raising=False) lock_mod._reset_for_tests() started: dict[str, Any] = {"engine": None} @@ -140,6 +142,7 @@ async def _setup(*, fake_llm: FakeLLMClient) -> None: # Silence OME strategies — orthogonal to boundary segmentation. mock_af = AsyncMock(return_value=[]) mock_fs = AsyncMock(return_value=[]) + mock_dc = AsyncMock(return_value=[]) monkeypatch.setattr( af_mod, "AtomicFactExtractor", @@ -150,6 +153,11 @@ async def _setup(*, fake_llm: FakeLLMClient) -> None: "ForesightExtractor", lambda *a, **k: type("M", (), {"aextract": mock_fs})(), ) + monkeypatch.setattr( + dc_mod, + "DecisionExtractor", + lambda *a, **k: type("M", (), {"aextract": mock_dc})(), + ) engine = svc._get_engine() await engine.start() diff --git a/tests/integration/test_ome_strategies_integration.py b/tests/integration/test_ome_strategies_integration.py index 88d9c8811..e642d00b8 100644 --- a/tests/integration/test_ome_strategies_integration.py +++ b/tests/integration/test_ome_strategies_integration.py @@ -106,8 +106,10 @@ async def test_emit_dispatches_both_strategies_to_success( monkeypatch.setattr(svc, "_ome_engine", None, raising=False) _af_mod = importlib.import_module("everos.memory.strategies.extract_atomic_facts") _fs_mod = importlib.import_module("everos.memory.strategies.extract_foresight") + _dc_mod = importlib.import_module("everos.memory.strategies.extract_decision") monkeypatch.setattr(_af_mod, "_writer", None, raising=False) monkeypatch.setattr(_fs_mod, "_writer", None, raising=False) + monkeypatch.setattr(_dc_mod, "_writer", None, raising=False) fake_fact = AtomicFact( owner_id="u_alice", content="hi", timestamp=1_700_000_000_000 @@ -126,6 +128,7 @@ async def test_emit_dispatches_both_strategies_to_success( patch( "everos.memory.strategies.extract_foresight.ForesightExtractor" ) as mock_fs, + patch("everos.memory.strategies.extract_decision.DecisionExtractor") as mock_dc, patch( "everos.memory.strategies.extract_atomic_facts.get_llm_client", return_value=object(), @@ -134,10 +137,15 @@ async def test_emit_dispatches_both_strategies_to_success( "everos.memory.strategies.extract_foresight.get_llm_client", return_value=object(), ), + patch( + "everos.memory.strategies.extract_decision.get_llm_client", + return_value=object(), + ), capture_logs() as logs, ): mock_af.return_value.aextract_from_text = AsyncMock(return_value=[fake_fact]) mock_fs.return_value.aextract = AsyncMock(return_value=[fake_foresight]) + mock_dc.return_value.aextract = AsyncMock(return_value=[]) # Ensure the sqlite dir exists before the engine creates ome.db. (tmp_path / ".index" / "sqlite").mkdir(parents=True, exist_ok=True) diff --git a/tests/integration/test_tiers/conftest.py b/tests/integration/test_tiers/conftest.py index b18e9d3f4..8a288eda6 100644 --- a/tests/integration/test_tiers/conftest.py +++ b/tests/integration/test_tiers/conftest.py @@ -100,7 +100,7 @@ def _boundary_response(boundaries: list[int]) -> str: def _episode_response(title: str = "Test Subject", content: str = "Test body") -> str: - return json.dumps({"title": title, "content": content}) + return json.dumps({"title": title, "content": content, "summary": content}) def make_fake_llm( @@ -114,7 +114,7 @@ def make_fake_llm( Mirrors ``tests/integration/test_memorize_integration.py``'s ``_make_fake_llm``: pops one ``boundaries=...`` entry per boundary prompt seen, every episode prompt gets the same canned - ``{title, content}``. Any other prompt (atomic facts / foresight / + ``{title, content, summary}``. Any other prompt (atomic facts / foresight / profile / agent case background strategies) also falls through to the episode-shaped response; those strategies run as OME background jobs that log-and-continue on a parse failure, so they never affect @@ -150,9 +150,12 @@ def handler(messages: list[LLMChatMessage], **_: Any) -> ChatResponse: _STRATEGY_SINGLETONS: tuple[tuple[str, tuple[str, ...]], ...] = ( ("everos.memory.strategies.extract_atomic_facts", ("_writer",)), ("everos.memory.strategies.extract_foresight", ("_writer",)), + ("everos.memory.strategies.extract_decision", ("_writer",)), ("everos.memory.strategies.extract_user_profile", ("_writer", "_reader")), + ("everos.memory.strategies.extract_principles", ("_writer", "_decision_reader")), ("everos.memory.strategies.extract_agent_case", ("_writer",)), ("everos.memory.strategies.extract_agent_skill", ("_writer",)), + ("everos.memory.strategies.reflect_decisions", ("_writer",)), ) diff --git a/tests/integration/test_tiers/test_tier1_keyword_only.py b/tests/integration/test_tiers/test_tier1_keyword_only.py index 9dacbb506..2321e7938 100644 --- a/tests/integration/test_tiers/test_tier1_keyword_only.py +++ b/tests/integration/test_tiers/test_tier1_keyword_only.py @@ -147,7 +147,7 @@ async def test_ome_registers_all_strategies_in_tier1( OME's registry is frozen after ``engine.start()`` (scheduler snapshots the strategy set for Cron / Idle job creation), so a registration-time - gate cannot adapt to a runtime tier upgrade. All eight strategies are + gate cannot adapt to a runtime tier upgrade. All nine strategies are registered up-front; the four embed-requiring ones check :func:`get_embedding_capability` at body entry and no-op when the capability is unavailable — see the per-strategy unit tests for that @@ -163,9 +163,13 @@ async def test_ome_registers_all_strategies_in_tier1( extract_agent_case, extract_agent_skill, extract_atomic_facts, + extract_decision, extract_foresight, + extract_principles, extract_user_profile, + reflect_decisions, reflect_episodes, + trigger_decision_clustering, trigger_profile_clustering, trigger_skill_clustering, ) @@ -174,13 +178,17 @@ async def test_ome_registers_all_strategies_in_tier1( strategy.meta.name for strategy in ( extract_atomic_facts, + extract_decision, extract_foresight, extract_agent_case, extract_user_profile, + extract_principles, trigger_profile_clustering, + trigger_decision_clustering, trigger_skill_clustering, extract_agent_skill, reflect_episodes, + reflect_decisions, ) } assert all_strategy_names <= registered_names diff --git a/tests/integration/test_tiers/test_tier2_no_rerank.py b/tests/integration/test_tiers/test_tier2_no_rerank.py index bb7574edd..04cc3234e 100644 --- a/tests/integration/test_tiers/test_tier2_no_rerank.py +++ b/tests/integration/test_tiers/test_tier2_no_rerank.py @@ -169,9 +169,13 @@ async def test_ome_registers_embed_dependent_strategies( extract_agent_case, extract_agent_skill, extract_atomic_facts, + extract_decision, extract_foresight, + extract_principles, extract_user_profile, + reflect_decisions, reflect_episodes, + trigger_decision_clustering, trigger_profile_clustering, trigger_skill_clustering, ) @@ -180,13 +184,17 @@ async def test_ome_registers_embed_dependent_strategies( strategy.meta.name for strategy in ( extract_atomic_facts, + extract_decision, extract_foresight, extract_agent_case, extract_user_profile, + extract_principles, trigger_profile_clustering, + trigger_decision_clustering, trigger_skill_clustering, extract_agent_skill, reflect_episodes, + reflect_decisions, ) } assert all_strategy_names <= registered_names diff --git a/tests/unit/test_entrypoints/test_api/test_routes/test_get_route_validation.py b/tests/unit/test_entrypoints/test_api/test_routes/test_get_route_validation.py index 11f0587b1..c69517e3e 100644 --- a/tests/unit/test_entrypoints/test_api/test_routes/test_get_route_validation.py +++ b/tests/unit/test_entrypoints/test_api/test_routes/test_get_route_validation.py @@ -91,7 +91,7 @@ async def test_missing_memory_type_returns_422(client: AsyncClient) -> None: async def test_invalid_memory_type_value_returns_422(client: AsyncClient) -> None: - """``memory_type`` outside the four-kind enum → 422.""" + """``memory_type`` outside the enumerated kinds → 422.""" resp = await client.post( "/api/v1/memory/get", json={ @@ -102,6 +102,18 @@ async def test_invalid_memory_type_value_returns_422(client: AsyncClient) -> Non assert resp.status_code == 422 +async def test_principle_memory_type_returns_422(client: AsyncClient) -> None: + """Principle is Meta Memory — not listable via ``/get``.""" + resp = await client.post( + "/api/v1/memory/get", + json={ + "user_id": "u1", + "memory_type": "principle", + }, + ) + assert resp.status_code == 422 + + async def test_invalid_sort_order_returns_422(client: AsyncClient) -> None: """``sort_order`` is a tight Literal — uppercase variant rejected.""" resp = await client.post( diff --git a/tests/unit/test_infra/test_lancedb/test_repos/test_decision.py b/tests/unit/test_infra/test_lancedb/test_repos/test_decision.py new file mode 100644 index 000000000..fa15b6bc5 --- /dev/null +++ b/tests/unit/test_infra/test_lancedb/test_repos/test_decision.py @@ -0,0 +1,25 @@ +"""Decision LanceDB repo can open an empty table.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from everos.infra.persistence.lancedb import decision_repo, lancedb_manager + + +@pytest.fixture +async def _real_lancedb(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + """Spin up a clean LanceDB rooted under ``tmp_path`` for one test.""" + monkeypatch.setenv("EVEROS_ROOT", str(tmp_path)) + lancedb_manager._conn = None + lancedb_manager._tables.clear() + yield + await lancedb_manager.dispose_connection() + + +async def test_repo_opens_empty_table(_real_lancedb: None) -> None: + table = await decision_repo._table() + assert await table.count_rows() == 0 + assert decision_repo.schema.TABLE_NAME == "decision" diff --git a/tests/unit/test_infra/test_lancedb/test_tables/test_content_sha256.py b/tests/unit/test_infra/test_lancedb/test_tables/test_content_sha256.py index ad8e40715..3ba93fdbb 100644 --- a/tests/unit/test_infra/test_lancedb/test_tables/test_content_sha256.py +++ b/tests/unit/test_infra/test_lancedb/test_tables/test_content_sha256.py @@ -16,6 +16,7 @@ AgentCase, AgentSkill, AtomicFact, + Decision, Episode, Foresight, ) @@ -82,6 +83,28 @@ def _foresight() -> Foresight: ) +def _decision() -> Decision: + return Decision( + id="u1_dc_1", + entry_id="dc_20260514_0001", + owner_id="u1", + owner_type="user", + session_id="s1", + timestamp=_NOW, + parent_type="memcell", + parent_id="mc_1", + title="Use Rust on device", + decision="Device Runtime uses Rust.", + reason="Need deterministic latency on the edge.", + tags=["runtime"], + decision_tokens="Device Runtime uses Rust", + reason_tokens="Need deterministic latency on the edge", + md_path="users/u1/decisions/decision-2026-05-14.md", + content_sha256=_SHA, + vector=_VEC, + ) + + def _agent_case() -> AgentCase: return AgentCase( id="a1_ac_1", @@ -124,8 +147,15 @@ def _agent_skill() -> AgentSkill: @pytest.mark.parametrize( "factory", - [_episode, _atomic_fact, _foresight, _agent_case, _agent_skill], - ids=["episode", "atomic_fact", "foresight", "agent_case", "agent_skill"], + [_episode, _atomic_fact, _decision, _foresight, _agent_case, _agent_skill], + ids=[ + "episode", + "atomic_fact", + "decision", + "foresight", + "agent_case", + "agent_skill", + ], ) def test_content_sha256_round_trip(factory) -> None: # type: ignore[no-untyped-def] row = factory() @@ -138,8 +168,15 @@ def test_content_sha256_round_trip(factory) -> None: # type: ignore[no-untyped- @pytest.mark.parametrize( "factory", - [_episode, _atomic_fact, _foresight, _agent_case, _agent_skill], - ids=["episode", "atomic_fact", "foresight", "agent_case", "agent_skill"], + [_episode, _atomic_fact, _decision, _foresight, _agent_case, _agent_skill], + ids=[ + "episode", + "atomic_fact", + "decision", + "foresight", + "agent_case", + "agent_skill", + ], ) def test_content_sha256_required(factory) -> None: # type: ignore[no-untyped-def] """Dropping content_sha256 from the kwargs surfaces a ValidationError.""" diff --git a/tests/unit/test_infra/test_lancedb/test_tables/test_decision.py b/tests/unit/test_infra/test_lancedb/test_tables/test_decision.py new file mode 100644 index 000000000..be42c9b49 --- /dev/null +++ b/tests/unit/test_infra/test_lancedb/test_tables/test_decision.py @@ -0,0 +1,91 @@ +"""Decision LanceDB table schema validation.""" + +from __future__ import annotations + +import datetime as dt + +import pytest +from pydantic import ValidationError + +from everos.infra.persistence.lancedb import Decision + + +def _kwargs(**overrides: object) -> dict[str, object]: + base: dict[str, object] = { + "id": "u_jason_dc_20260826_0001", + "entry_id": "dc_20260826_0001", + "owner_id": "u_jason", + "owner_type": "user", + "timestamp": dt.datetime(2026, 8, 26, 12, 0, tzinfo=dt.UTC), + "parent_id": "mc_1", + "title": "Use Rust on device", + "decision": "Device Runtime uses Rust.", + "reason": "Need deterministic latency on the edge.", + "tags": ["runtime", "rust"], + "decision_tokens": "Device Runtime uses Rust", + "reason_tokens": "Need deterministic latency on the edge", + "md_path": "users/u_jason/decisions/decision-2026-08-26.md", + "content_sha256": "a" * 64, + } + base.update(overrides) + return base + + +class TestDecisionSchema: + def test_table_name(self) -> None: + assert Decision.TABLE_NAME == "decision" + + def test_bm25_fields_dual_column(self) -> None: + assert Decision.BM25_FIELDS == ["decision_tokens", "reason_tokens"] + + def test_has_required_fields(self) -> None: + fields = set(Decision.model_fields.keys()) + required = { + "id", + "entry_id", + "owner_id", + "owner_type", + "app_id", + "project_id", + "session_id", + "timestamp", + "parent_type", + "parent_id", + "title", + "decision", + "reason", + "impact", + "tags", + "decision_tokens", + "reason_tokens", + "md_path", + "content_sha256", + "vector", + "deprecated_by", + } + assert required.issubset(fields), f"Missing: {required - fields}" + assert "sender_ids" not in fields + + def test_vector_nullable_at_pyarrow_layer(self) -> None: + arrow = Decision.to_arrow_schema() + assert arrow.field("vector").nullable is True + + def test_constructs_minimal_row(self) -> None: + row = Decision(**_kwargs()) # type: ignore[arg-type] + assert row.impact is None + assert row.deprecated_by is None + assert row.vector is None + assert row.parent_type == "memcell" + assert row.tags == ["runtime", "rust"] + + def test_missing_decision_raises(self) -> None: + bad = _kwargs() + del bad["decision"] + with pytest.raises(ValidationError): + Decision(**bad) # type: ignore[arg-type] + + def test_missing_content_sha256_raises(self) -> None: + bad = _kwargs() + del bad["content_sha256"] + with pytest.raises(ValidationError): + Decision(**bad) # type: ignore[arg-type] diff --git a/tests/unit/test_infra/test_lancedb_tables_nullability.py b/tests/unit/test_infra/test_lancedb_tables_nullability.py index 202294767..a63dc7516 100644 --- a/tests/unit/test_infra/test_lancedb_tables_nullability.py +++ b/tests/unit/test_infra/test_lancedb_tables_nullability.py @@ -23,6 +23,7 @@ AgentCase, AgentSkill, AtomicFact, + Decision, Episode, Foresight, KnowledgeTopic, @@ -71,22 +72,28 @@ def test_knowledge_topic_vector_nullable(): assert "None" in ann +def test_decision_vector_nullable(): + ann = str(_vector_field_type(Decision)) + assert "None" in ann + + def test_dim_unchanged(): """_DIM stays at 1024 across all table modules — only nullability moves.""" from everos.infra.persistence.lancedb.tables import agent_case as m1 from everos.infra.persistence.lancedb.tables import agent_skill as m2 from everos.infra.persistence.lancedb.tables import atomic_fact as m3 - from everos.infra.persistence.lancedb.tables import episode as m4 - from everos.infra.persistence.lancedb.tables import foresight as m5 - from everos.infra.persistence.lancedb.tables import knowledge_topic as m6 + from everos.infra.persistence.lancedb.tables import decision as m4 + from everos.infra.persistence.lancedb.tables import episode as m5 + from everos.infra.persistence.lancedb.tables import foresight as m6 + from everos.infra.persistence.lancedb.tables import knowledge_topic as m7 - for module in (m1, m2, m3, m4, m5, m6): + for module in (m1, m2, m3, m4, m5, m6, m7): assert module._DIM == 1024 @pytest.mark.parametrize( "cls", - [Episode, AtomicFact, Foresight, AgentCase, AgentSkill, KnowledgeTopic], + [Episode, AtomicFact, Decision, Foresight, AgentCase, AgentSkill, KnowledgeTopic], ) def test_vector_nullable_at_pyarrow_layer(cls): """``alter_columns`` (the migration primitive) operates on the pyarrow diff --git a/tests/unit/test_infra/test_markdown/test_mds/test_decision.py b/tests/unit/test_infra/test_markdown/test_mds/test_decision.py new file mode 100644 index 000000000..1c44e7fdb --- /dev/null +++ b/tests/unit/test_infra/test_markdown/test_mds/test_decision.py @@ -0,0 +1,81 @@ +"""Tests for :class:`DecisionDailyFrontmatter`. + +Lives under ``test_infra`` because the schema lives under +``infra/.../mds``. Outer layers must import from the package top-level +(``everos.infra.persistence.markdown``), not ``markdown.mds``. +""" + +from __future__ import annotations + +import datetime as _dt + +import pytest +from pydantic import ValidationError + +from everos.infra.persistence.markdown import DecisionDailyFrontmatter + + +def _kwargs(**overrides: object) -> dict[str, object]: + """Minimal valid kwargs for DecisionDailyFrontmatter.""" + base: dict[str, object] = { + "id": "decision_daily_u_jason_2026-08-26", + "user_id": "u_jason", + "date": _dt.date(2026, 8, 26), + } + base.update(overrides) + return base + + +def test_classvars() -> None: + assert DecisionDailyFrontmatter.ENTRY_ID_PREFIX == "dc" + assert DecisionDailyFrontmatter.DIR_NAME == "decisions" + assert DecisionDailyFrontmatter.FILE_PREFIX == "decision" + assert DecisionDailyFrontmatter.SCOPE_DIR == "users" + + +def test_dir_name_is_not_dot_prefixed() -> None: + """Decisions are user-readable, unlike ``.atomic_facts`` / ``.foresights``.""" + assert not DecisionDailyFrontmatter.DIR_NAME.startswith(".") + + +def test_entry_id_prefix_is_token_not_full_prefix() -> None: + """Chassis mints ``dc__`` from the token ``dc``, not ``dc_``.""" + assert DecisionDailyFrontmatter.ENTRY_ID_PREFIX == "dc" + assert "_" not in DecisionDailyFrontmatter.ENTRY_ID_PREFIX + + +def test_constructs_with_id_user_id_date() -> None: + fm = DecisionDailyFrontmatter(**_kwargs()) # type: ignore[arg-type] + assert fm.id == "decision_daily_u_jason_2026-08-26" + assert fm.user_id == "u_jason" + assert fm.date == _dt.date(2026, 8, 26) + assert fm.type == "decision_daily" + assert fm.file_type == "decision_daily" + assert fm.track == "user" + assert fm.entry_count == 0 + + +def test_deprecated_entries_defaults_empty() -> None: + fm = DecisionDailyFrontmatter(**_kwargs()) # type: ignore[arg-type] + assert fm.deprecated_entries == {} + + +def test_missing_date_raises() -> None: + bad = _kwargs() + del bad["date"] + with pytest.raises(ValidationError): + DecisionDailyFrontmatter(**bad) # type: ignore[arg-type] + + +def test_deprecated_entries_round_trip() -> None: + fm = DecisionDailyFrontmatter( + **_kwargs( + deprecated_entries={"dc_20260826_001": "dc_20260826_002"}, + ), # type: ignore[arg-type] + ) + dumped = fm.model_dump() + assert dumped["deprecated_entries"] == {"dc_20260826_001": "dc_20260826_002"} + round_tripped = DecisionDailyFrontmatter.model_validate(dumped) + assert round_tripped.deprecated_entries == { + "dc_20260826_001": "dc_20260826_002", + } diff --git a/tests/unit/test_infra/test_markdown/test_mds/test_path_glob.py b/tests/unit/test_infra/test_markdown/test_mds/test_path_glob.py index 17ba54537..089dbfd37 100644 --- a/tests/unit/test_infra/test_markdown/test_mds/test_path_glob.py +++ b/tests/unit/test_infra/test_markdown/test_mds/test_path_glob.py @@ -11,6 +11,7 @@ AgentCaseDailyFrontmatter, AgentSkillFrontmatter, AtomicFactDailyFrontmatter, + DecisionDailyFrontmatter, EpisodeDailyFrontmatter, ForesightDailyFrontmatter, ) @@ -20,6 +21,7 @@ ("schema", "expected"), [ (EpisodeDailyFrontmatter, "*/*/users/*/episodes/episode-*.md"), + (DecisionDailyFrontmatter, "*/*/users/*/decisions/decision-*.md"), (AtomicFactDailyFrontmatter, "*/*/users/*/.atomic_facts/atomic_fact-*.md"), (ForesightDailyFrontmatter, "*/*/users/*/.foresights/foresight-*.md"), (AgentCaseDailyFrontmatter, "*/*/agents/*/.cases/agent_case-*.md"), diff --git a/tests/unit/test_infra/test_markdown/test_mds/test_principle.py b/tests/unit/test_infra/test_markdown/test_mds/test_principle.py new file mode 100644 index 000000000..895ee5bb7 --- /dev/null +++ b/tests/unit/test_infra/test_markdown/test_mds/test_principle.py @@ -0,0 +1,105 @@ +"""Tests for :class:`PrincipleFrontmatter` and ProfileWriter reuse.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from everos.core.persistence import MemoryRoot +from everos.infra.persistence.markdown import ( + PrincipleFrontmatter, + PrincipleItem, + ProfileReader, + ProfileWriter, + mint_principle_id, + render_principles_body, +) + + +def test_mint_principle_id_shape() -> None: + pid = mint_principle_id() + assert pid.startswith("pr_") + assert len(pid) == 15 + assert pid != mint_principle_id() + + +def test_schema_pins_profile_chassis() -> None: + assert PrincipleFrontmatter.PROFILE_FILENAME == "principles.md" + assert PrincipleFrontmatter.SCOPE_DIR == "users" + assert PrincipleFrontmatter.path_glob() == "*/*/users/*/principles.md" + + +def test_type_defaults_to_principle() -> None: + fm = PrincipleFrontmatter(id="principle_u_alice", user_id="u_alice") + assert fm.type == "principle" + assert fm.track == "user" + assert fm.principles == [] + + +def test_nested_item_defaults() -> None: + item = PrincipleItem( + id="pr_aaaaaaaaaaaa", + title="Use Rust on device", + statement="Device Runtime uses Rust.", + ) + assert item.source_entry_ids == [] + assert item.timestamp_ms == 0 + + +def test_item_requires_id_title_statement() -> None: + with pytest.raises(ValidationError): + PrincipleItem(id="pr_1", title="T") # type: ignore[call-arg] + + +def test_render_principles_body() -> None: + items = [ + PrincipleItem( + id="pr_aaaaaaaaaaaa", + title="Use Rust on device", + statement="Device Runtime uses Rust.", + ) + ] + assert render_principles_body(items) == ( + "- **Use Rust on device.** Device Runtime uses Rust.\n" + ) + assert render_principles_body([]) == "" + + +async def test_profile_writer_round_trip(tmp_path: Path) -> None: + """Principles reuse ProfileWriter — no fourth storage strategy.""" + root = MemoryRoot(tmp_path) + writer = ProfileWriter(root) + reader = ProfileReader(root) + items = [ + PrincipleItem( + id="pr_aaaaaaaaaaaa", + title="Use Rust on device", + statement="Device Runtime uses Rust.", + source_entry_ids=["dc_20260517_0001"], + timestamp_ms=1_700_000_000_000, + ) + ] + fm = PrincipleFrontmatter( + id="principle_u_alice", + user_id="u_alice", + principles=items, + ) + path = await writer.write( + "u_alice", + frontmatter=fm, + body=render_principles_body(items), + ) + expected = root.users_dir() / "u_alice" / "principles.md" + assert path == expected + assert expected.is_file() + + out = await reader.read("u_alice", schema=PrincipleFrontmatter) + assert out is not None + fm_out, body = out + assert fm_out.type == "principle" + assert len(fm_out.principles) == 1 + assert fm_out.principles[0].id == "pr_aaaaaaaaaaaa" + assert fm_out.principles[0].source_entry_ids == ["dc_20260517_0001"] + assert "Use Rust on device" in body diff --git a/tests/unit/test_infra/test_markdown/test_writers/test_daily_log_writers.py b/tests/unit/test_infra/test_markdown/test_writers/test_daily_log_writers.py index 20372fb37..191255ac2 100644 --- a/tests/unit/test_infra/test_markdown/test_writers/test_daily_log_writers.py +++ b/tests/unit/test_infra/test_markdown/test_writers/test_daily_log_writers.py @@ -1,6 +1,6 @@ -"""Tests for AtomicFact / Foresight / AgentCase daily-log writers. +"""Tests for AtomicFact / Decision / Foresight / AgentCase daily-log writers. -The 4 daily-log kinds (episode + these 3) all share ``BaseDailyWriter`` +The 5 daily-log kinds (episode + these 4) all share ``BaseDailyWriter`` plumbing — exhaustive chassis tests live in ``test_base.py`` and ``test_episode_writer.py`` indirectly via the e2e flows. Here we focus on the per-kind path resolution + frontmatter shape that each @@ -21,6 +21,8 @@ AgentCaseWriter, AtomicFactReader, AtomicFactWriter, + DecisionReader, + DecisionWriter, ForesightReader, ForesightWriter, ) @@ -112,6 +114,98 @@ async def test_atomic_fact_writer_appends_multiple(memory_root: MemoryRoot) -> N assert eid2.format().endswith("0002") +# ── Decision ────────────────────────────────────────────────────────────── + + +def _decision_inline(**overrides: object) -> dict[str, object]: + base: dict[str, object] = { + "owner_id": "u1", + "session_id": "s1", + "timestamp": "2026-05-15T10:00:00+00:00", + "parent_type": "memcell", + "parent_id": "mc_1", + "tags": ["runtime", "rust"], + } + base.update(overrides) + return base + + +def _decision_sections(**overrides: str) -> dict[str, str]: + base = { + "Title": "Use Rust on device", + "Decision": "Device Runtime uses Rust.", + "Reason": "Need deterministic latency on the edge.", + "Impact": "Keep Python in the agent runtime.", + } + base.update(overrides) + return base + + +async def test_decision_writer_round_trip(memory_root: MemoryRoot) -> None: + writer = DecisionWriter(memory_root) + today = _dt.date(2026, 5, 15) + eid = await writer.append_entry( + "u1", + inline=_decision_inline(), + sections=_decision_sections(), + date=today, + ) + path = memory_root.users_dir() / "u1" / "decisions" / "decision-2026-05-15.md" + parsed = await MarkdownReader.read(path) + + fm = parsed.frontmatter + assert fm["id"] == "decision_log_u1_2026-05-15" + assert fm["type"] == "decision_daily" + assert fm["file_type"] == "decision_daily" + assert fm["user_id"] == "u1" + assert fm["track"] == "user" + assert fm["date"] == "2026-05-15" + assert fm["entry_count"] == 1 + + assert eid.format().startswith("dc_") + assert eid.format() == "dc_20260515_00000001" + assert len(parsed.entries) == 1 + structured = parsed.entries[0].as_structured() + assert structured.inline["owner_id"] == "u1" + assert structured.inline["parent_type"] == "memcell" + assert structured.inline["parent_id"] == "mc_1" + assert "sender_ids" not in structured.inline + assert "runtime" in structured.inline["tags"] + assert structured.sections["Title"] == "Use Rust on device" + assert structured.sections["Decision"] == "Device Runtime uses Rust." + assert structured.sections["Reason"] == "Need deterministic latency on the edge." + assert structured.sections["Impact"] == "Keep Python in the agent runtime." + + reader = DecisionReader(memory_root) + assert reader.path_for("u1", today) == path + found = await reader.find_structured("u1", eid) + assert found is not None + assert found.sections["Decision"] == "Device Runtime uses Rust." + + +async def test_decision_writer_appends_multiple(memory_root: MemoryRoot) -> None: + writer = DecisionWriter(memory_root) + today = _dt.date(2026, 5, 15) + eid1 = await writer.append_entry( + "u1", + inline=_decision_inline(), + sections=_decision_sections(), + date=today, + ) + eid2 = await writer.append_entry( + "u1", + inline=_decision_inline( + parent_id="mc_2", + timestamp="2026-05-15T11:00:00+00:00", + ), + sections=_decision_sections(Title="Second", Decision="Keep local-first."), + date=today, + ) + assert eid1.format() != eid2.format() + assert eid1.format().startswith("dc_") + assert eid2.format().endswith("00000002") + + # ── Foresight ───────────────────────────────────────────────────────────── @@ -279,9 +373,9 @@ async def test_atomic_fact_frontmatter_last_appended_at_carries_display_tz_offse write an entry, read the .md file, assert the literal string ends with ``+08:00``. - Repeats the same check for ``ForesightWriter`` and - ``AgentCaseWriter`` — they share ``BaseDailyWriter`` plumbing so a - regression on one would likely affect all three, but pinning each + Repeats the same check for ``ForesightWriter``, ``DecisionWriter``, + and ``AgentCaseWriter`` — they share ``BaseDailyWriter`` plumbing so a + regression on one would likely affect all of them, but pinning each rules out per-subclass shadowing of ``_frontmatter_updates``. """ from everos.component.utils import datetime as _dt_module @@ -331,6 +425,18 @@ async def test_atomic_fact_frontmatter_last_appended_at_carries_display_tz_offse fs_fm = (await MarkdownReader.read(fs_path)).frontmatter assert fs_fm["last_appended_at"].endswith("+08:00"), fs_fm["last_appended_at"] + # Decision + dc_writer = DecisionWriter(memory_root) + await dc_writer.append_entry( + "u1", + inline=_decision_inline(), + sections=_decision_sections(), + date=today, + ) + dc_path = memory_root.users_dir() / "u1" / "decisions" / "decision-2026-05-15.md" + dc_fm = (await MarkdownReader.read(dc_path)).frontmatter + assert dc_fm["last_appended_at"].endswith("+08:00"), dc_fm["last_appended_at"] + # AgentCase ac_writer = AgentCaseWriter(memory_root) await ac_writer.append_entry( diff --git a/tests/unit/test_memory/test_cascade/test_backfill_engine_isolation.py b/tests/unit/test_memory/test_cascade/test_backfill_engine_isolation.py index 1fc3ab6bb..980750ead 100644 --- a/tests/unit/test_memory/test_cascade/test_backfill_engine_isolation.py +++ b/tests/unit/test_memory/test_cascade/test_backfill_engine_isolation.py @@ -99,7 +99,7 @@ async def test_backfill_engine_start_does_not_reenqueue_stale_running_rows( ) # And no APS job was scheduled for the stale run_id — the # backfill scheduler stayed empty (only Cron/Idle jobs would - # exist, and neither cluster strategy uses those triggers). + # exist, and none of the cluster strategies use those triggers). assert engine._scheduler.get_job("r_server_stale") is None finally: await engine.stop() diff --git a/tests/unit/test_memory/test_cascade/test_backfill_phase2_idempotency.py b/tests/unit/test_memory/test_cascade/test_backfill_phase2_idempotency.py index 72346e04e..3d82f5f96 100644 --- a/tests/unit/test_memory/test_cascade/test_backfill_phase2_idempotency.py +++ b/tests/unit/test_memory/test_cascade/test_backfill_phase2_idempotency.py @@ -15,8 +15,9 @@ it toward the progress readout (matches the pre-scan estimate). The ``member_type`` values used for the lookup (``"episode"`` / -``"case"``) must match what :func:`trigger_profile_clustering` / -:func:`trigger_skill_clustering` insert on the write path — a +``"case"`` / ``"decision"``) must match what +:func:`trigger_profile_clustering` / :func:`trigger_skill_clustering` / +:func:`trigger_decision_clustering` insert on the write path — a mismatch would silently disable the dedup. """ @@ -27,7 +28,7 @@ from everos.component.utils.datetime import get_utc_now from everos.memory.cascade import _backfill from everos.memory.cascade._backfill import NullBackfillPresenter -from everos.memory.events import AgentCaseExtracted, EpisodeExtracted +from everos.memory.events import AgentCaseExtracted, DecisionExtracted, EpisodeExtracted class _RecordingEngine: @@ -66,6 +67,23 @@ def _case_row(entry_id: str) -> dict[str, Any]: } +def _decision_row(entry_id: str) -> dict[str, Any]: + return { + "entry_id": entry_id, + "parent_id": f"mc_{entry_id}", + "title": f"title {entry_id}", + "decision": f"body {entry_id}", + "reason": f"reason {entry_id}", + "impact": None, + "tags": ["runtime"], + "timestamp": get_utc_now(), + "owner_id": "u1", + "session_id": "s1", + "app_id": "default", + "project_id": "default", + } + + async def test_emit_synthetic_events_skips_already_clustered_rows( monkeypatch, ) -> None: @@ -76,6 +94,7 @@ async def test_emit_synthetic_events_skips_already_clustered_rows( clustered = { ("episode", "ep_dup"): "cluster_abc", ("case", "ac_dup"): "cluster_def", + ("decision", "dc_dup"): "cluster_ghi", } class _StubClusterRepo: @@ -96,9 +115,14 @@ async def find_cluster_id_for_member( engine = _RecordingEngine() episodes = [_episode_row("ep_fresh"), _episode_row("ep_dup")] cases = [_case_row("ac_fresh"), _case_row("ac_dup")] + decisions = [_decision_row("dc_fresh"), _decision_row("dc_dup")] processed = await _backfill._emit_synthetic_events( - engine, episodes, cases, presenter=NullBackfillPresenter() + engine, + episodes, + cases, + presenter=NullBackfillPresenter(), + decisions=decisions, ) # Every row was looked up — dedup must not depend on ordering. @@ -107,32 +131,38 @@ async def find_cluster_id_for_member( ("episode", "ep_dup"), ("case", "ac_fresh"), ("case", "ac_dup"), + ("decision", "dc_fresh"), + ("decision", "dc_dup"), } - # Only the fresh rows produced an engine.emit call — the two - # already-clustered rows were skipped. - assert len(engine.emitted) == 2 + # Only the fresh rows produced an engine.emit call — the already- + # clustered rows were skipped. + assert len(engine.emitted) == 3 episode_ids = { e.episode_entry_id for e in engine.emitted if isinstance(e, EpisodeExtracted) } case_ids = { e.case_entry_id for e in engine.emitted if isinstance(e, AgentCaseExtracted) } + decision_ids = { + e.decision_entry_id for e in engine.emitted if isinstance(e, DecisionExtracted) + } assert episode_ids == {"ep_fresh"} assert case_ids == {"ac_fresh"} + assert decision_ids == {"dc_fresh"} # Progress count still equals the full input size so the readout # matches the pre-scan estimate the user just confirmed. - assert processed == 4 + assert processed == 6 async def test_emit_synthetic_events_uses_write_path_member_type_strings( monkeypatch, ) -> None: """Regression guard: the ``member_type`` strings on the lookup path - (``"episode"`` / ``"case"``) must match what the clustering - strategies persist on the write path. A drift here would silently - disable the dedup and re-open the double-cluster window.""" + (``"episode"`` / ``"case"`` / ``"decision"``) must match what the + clustering strategies persist on the write path. A drift here would + silently disable the dedup and re-open the double-cluster window.""" seen_member_types: set[str] = set() class _StubClusterRepo: @@ -156,12 +186,14 @@ async def find_cluster_id_for_member( [_episode_row("ep1")], [_case_row("ac1")], presenter=NullBackfillPresenter(), + decisions=[_decision_row("dc1")], ) # These string constants are pinned by - # ``trigger_profile_clustering`` (member_type="episode") and - # ``trigger_skill_clustering`` (member_type="case"). - assert seen_member_types == {"episode", "case"} + # ``trigger_profile_clustering`` (member_type="episode"), + # ``trigger_skill_clustering`` (member_type="case"), and + # ``trigger_decision_clustering`` (member_type="decision"). + assert seen_member_types == {"episode", "case", "decision"} async def test_emit_synthetic_events_no_skip_when_no_prior_clusters( @@ -190,7 +222,9 @@ async def find_cluster_id_for_member( [_episode_row("ep1"), _episode_row("ep2")], [_case_row("ac1")], presenter=NullBackfillPresenter(), + decisions=[_decision_row("dc1")], ) - assert processed == 3 - assert len(engine.emitted) == 3 + assert processed == 4 + assert len(engine.emitted) == 4 + assert any(isinstance(e, DecisionExtracted) for e in engine.emitted) diff --git a/tests/unit/test_memory/test_cascade/test_backfill_preflight.py b/tests/unit/test_memory/test_cascade/test_backfill_preflight.py index 706201273..bc6d638cc 100644 --- a/tests/unit/test_memory/test_cascade/test_backfill_preflight.py +++ b/tests/unit/test_memory/test_cascade/test_backfill_preflight.py @@ -614,7 +614,7 @@ async def test_phase_clusters_preflight_before_scan( _isolated_root: Path, monkeypatch ) -> None: """Phase 2 mirrors Phase 1: capability preflight fires before - ``_scan_all_rows`` walks Episode + AgentCase.""" + ``_scan_all_rows`` walks Episode + AgentCase + Decision.""" monkeypatch.setattr( _backfill, "get_embedding_capability", lambda: _FakeCapabilityMissing() ) diff --git a/tests/unit/test_memory/test_cascade/test_backfill_table_specs.py b/tests/unit/test_memory/test_cascade/test_backfill_table_specs.py index d4ca1ea17..2e8891704 100644 --- a/tests/unit/test_memory/test_cascade/test_backfill_table_specs.py +++ b/tests/unit/test_memory/test_cascade/test_backfill_table_specs.py @@ -33,6 +33,20 @@ def test_table_specs_covers_business_schemas() -> None: spec_names = {spec.schema.TABLE_NAME for spec in _backfill._TABLE_SPECS} schema_names = {schema.TABLE_NAME for schema in BUSINESS_SCHEMAS_WITH_VECTOR} assert spec_names == schema_names + assert "principle" not in spec_names + assert "user_profile" not in spec_names + + +def test_principle_is_business_schema_without_vector() -> None: + """Principle is created at startup (``_BUSINESS_SCHEMAS``) but is KV + only — no Phase-1 vector backfill, no BM25.""" + from everos.infra.persistence.lancedb import _BUSINESS_SCHEMAS, Principle + + assert Principle in _BUSINESS_SCHEMAS + assert Principle.BM25_FIELDS == [] + assert "vector" not in Principle.model_fields + vector_names = {schema.TABLE_NAME for schema in BUSINESS_SCHEMAS_WITH_VECTOR} + assert "principle" not in vector_names def test_drift_scenario_actually_raises_at_import() -> None: diff --git a/tests/unit/test_memory/test_cascade/test_handler_principle.py b/tests/unit/test_memory/test_cascade/test_handler_principle.py new file mode 100644 index 000000000..4f22171f3 --- /dev/null +++ b/tests/unit/test_memory/test_cascade/test_handler_principle.py @@ -0,0 +1,280 @@ +"""Tests for :class:`PrincipleHandler` — explode one file into N rows.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from everos.component.tokenizer import Tokenizer +from everos.core.persistence import MemoryRoot +from everos.infra.persistence.lancedb import Principle +from everos.infra.persistence.markdown import ( + PrincipleFrontmatter, + PrincipleItem, + ProfileWriter, + render_principles_body, +) +from everos.memory.cascade.handlers import HandlerDeps, PrincipleHandler + + +class _StubTokenizer(Tokenizer): + def tokenize(self, text: str) -> list[str]: + return [tok for tok in text.split() if tok] + + def tokenize_batch(self, texts): # type: ignore[no-untyped-def] + return [self.tokenize(t) for t in texts] + + +class _FakePrincipleRepo: + def __init__(self) -> None: + self.rows: dict[str, Principle] = {} + self.upserts: list[list[Principle]] = [] + self.delete_predicates: list[str] = [] + self.deletes_by_md: list[str] = [] + + async def find_where(self, where: str, *, limit: int = 100) -> list[Principle]: + md_path = where.split("md_path = '", 1)[1].rsplit("'", 1)[0] + found = [r for r in self.rows.values() if r.md_path == md_path] + return found[:limit] + + async def upsert(self, rows: list[Principle]) -> None: + self.upserts.append(list(rows)) + for row in rows: + self.rows[row.id] = row + + async def delete(self, predicate: str) -> None: + self.delete_predicates.append(predicate) + inside = predicate.split(" IN (", 1)[1].rstrip(")") + ids = [tok.strip().strip("'") for tok in inside.split(",")] + for row_id in ids: + self.rows.pop(row_id, None) + + async def delete_by_md_path(self, md_path: str) -> int: + self.deletes_by_md.append(md_path) + before = len(self.rows) + self.rows = {rid: r for rid, r in self.rows.items() if r.md_path != md_path} + return before - len(self.rows) + + +@pytest.fixture +def memory_root(tmp_path: Path) -> MemoryRoot: + mr = MemoryRoot(tmp_path) + mr.ensure() + return mr + + +@pytest.fixture +def fake_repo(monkeypatch: pytest.MonkeyPatch) -> _FakePrincipleRepo: + from everos.memory.cascade.handlers import principle as pr_mod + + repo = _FakePrincipleRepo() + monkeypatch.setattr(pr_mod, "principle_repo", repo) + return repo + + +async def _write_principles( + memory_root: MemoryRoot, + user_id: str, + items: list[PrincipleItem], +) -> str: + writer = ProfileWriter(memory_root) + fm = PrincipleFrontmatter( + id=f"principle_{user_id}", + user_id=user_id, + principles=items, + ) + await writer.write( + user_id, + frontmatter=fm, + body=render_principles_body(items), + ) + return f"default_app/default_project/users/{user_id}/principles.md" + + +def _handler(memory_root: MemoryRoot) -> PrincipleHandler: + return PrincipleHandler( + HandlerDeps( + memory_root=memory_root, + tokenizer=_StubTokenizer(), + ) + ) + + +def _item( + principle_id: str, + *, + title: str = "Use Rust on device", + statement: str = "Device Runtime uses Rust.", + source_entry_ids: list[str] | None = None, + timestamp_ms: int = 1_700_000_000_000, +) -> PrincipleItem: + return PrincipleItem( + id=principle_id, + title=title, + statement=statement, + source_entry_ids=source_entry_ids or ["dc_20260517_0001"], + timestamp_ms=timestamp_ms, + ) + + +async def test_first_pass_explodes_into_n_rows( + memory_root: MemoryRoot, fake_repo: _FakePrincipleRepo +) -> None: + md_path = await _write_principles( + memory_root, + "u_alice", + [_item("pr_aaaaaaaaaaaa"), _item("pr_bbbbbbbbbbbb", title="Keep Python")], + ) + outcome = await _handler(memory_root).handle_added_or_modified(md_path) + + assert outcome.kind == "principle" + assert outcome.upserted == 2 + assert outcome.deleted == 0 + assert outcome.skipped == 0 + rows = fake_repo.upserts[0] + assert {r.id for r in rows} == { + "u_alice_pr_aaaaaaaaaaaa", + "u_alice_pr_bbbbbbbbbbbb", + } + rust = next(r for r in rows if r.principle_id == "pr_aaaaaaaaaaaa") + assert rust.owner_id == "u_alice" + assert rust.owner_type == "user" + assert rust.title == "Use Rust on device" + assert rust.statement == "Device Runtime uses Rust." + assert rust.source_entry_ids == ["dc_20260517_0001"] + assert rust.md_path == md_path + assert "vector" not in Principle.model_fields + + +async def test_second_pass_with_same_content_skips( + memory_root: MemoryRoot, fake_repo: _FakePrincipleRepo +) -> None: + md_path = await _write_principles( + memory_root, "u_alice", [_item("pr_aaaaaaaaaaaa")] + ) + handler = _handler(memory_root) + first = await handler.handle_added_or_modified(md_path) + assert first.upserted == 1 + + second = await handler.handle_added_or_modified(md_path) + assert second.upserted == 0 + assert second.skipped == 1 + assert second.deleted == 0 + assert len(fake_repo.upserts) == 1 + + +async def test_timestamp_only_drift_skips( + memory_root: MemoryRoot, fake_repo: _FakePrincipleRepo +) -> None: + md_path = await _write_principles( + memory_root, + "u_alice", + [_item("pr_aaaaaaaaaaaa", timestamp_ms=1_700_000_000_000)], + ) + handler = _handler(memory_root) + await handler.handle_added_or_modified(md_path) + + absolute = memory_root.root / md_path + absolute.write_text( + absolute.read_text(encoding="utf-8").replace("1700000000000", "1800000000000"), + encoding="utf-8", + ) + outcome = await handler.handle_added_or_modified(md_path) + assert outcome.upserted == 0 + assert outcome.skipped == 1 + + +async def test_removed_item_deletes_row( + memory_root: MemoryRoot, fake_repo: _FakePrincipleRepo +) -> None: + md_path = await _write_principles( + memory_root, + "u_alice", + [_item("pr_aaaaaaaaaaaa"), _item("pr_bbbbbbbbbbbb", title="Keep Python")], + ) + handler = _handler(memory_root) + await handler.handle_added_or_modified(md_path) + assert len(fake_repo.rows) == 2 + + md_path = await _write_principles( + memory_root, "u_alice", [_item("pr_aaaaaaaaaaaa")] + ) + outcome = await handler.handle_added_or_modified(md_path) + assert outcome.upserted == 0 + assert outcome.deleted == 1 + assert outcome.skipped == 1 + assert list(fake_repo.rows) == ["u_alice_pr_aaaaaaaaaaaa"] + assert fake_repo.delete_predicates + assert "u_alice_pr_bbbbbbbbbbbb" in fake_repo.delete_predicates[0] + + +async def test_statement_edit_triggers_upsert( + memory_root: MemoryRoot, fake_repo: _FakePrincipleRepo +) -> None: + md_path = await _write_principles( + memory_root, "u_alice", [_item("pr_aaaaaaaaaaaa")] + ) + handler = _handler(memory_root) + await handler.handle_added_or_modified(md_path) + + md_path = await _write_principles( + memory_root, + "u_alice", + [_item("pr_aaaaaaaaaaaa", statement="Prefer Rust for device Runtime.")], + ) + outcome = await handler.handle_added_or_modified(md_path) + assert outcome.upserted == 1 + assert fake_repo.upserts[1][0].statement == "Prefer Rust for device Runtime." + + +async def test_missing_user_id_raises( + memory_root: MemoryRoot, fake_repo: _FakePrincipleRepo +) -> None: + bad_dir = memory_root.root / "default_app" / "default_project" / "users" / "u_x" + bad_dir.mkdir(parents=True, exist_ok=True) + (bad_dir / "principles.md").write_text( + "---\nid: principle_u_x\ntype: principle\ntrack: user\nprinciples: []\n---\n", + encoding="utf-8", + ) + + with pytest.raises(ValueError, match="user_id"): + await _handler(memory_root).handle_added_or_modified( + "default_app/default_project/users/u_x/principles.md" + ) + + +async def test_duplicate_ids_raise( + memory_root: MemoryRoot, fake_repo: _FakePrincipleRepo +) -> None: + md_path = await _write_principles( + memory_root, + "u_alice", + [_item("pr_aaaaaaaaaaaa"), _item("pr_aaaaaaaaaaaa", title="Dup")], + ) + with pytest.raises(ValueError, match="duplicate id"): + await _handler(memory_root).handle_added_or_modified(md_path) + + +async def test_handle_deleted_drops_all_rows( + memory_root: MemoryRoot, fake_repo: _FakePrincipleRepo +) -> None: + md_path = await _write_principles( + memory_root, + "u_alice", + [_item("pr_aaaaaaaaaaaa"), _item("pr_bbbbbbbbbbbb", title="Keep Python")], + ) + handler = _handler(memory_root) + await handler.handle_added_or_modified(md_path) + assert len(fake_repo.rows) == 2 + + outcome = await handler.handle_deleted(md_path) + assert outcome.deleted == 2 + assert fake_repo.deletes_by_md == [md_path] + assert fake_repo.rows == {} + + +def test_lance_schema_has_no_vector_or_bm25() -> None: + assert Principle.TABLE_NAME == "principle" + assert Principle.BM25_FIELDS == [] + assert "vector" not in Principle.model_fields diff --git a/tests/unit/test_memory/test_cascade/test_handlers_daily_log_mapping.py b/tests/unit/test_memory/test_cascade/test_handlers_daily_log_mapping.py index b3fcb4a46..6aebbf3f8 100644 --- a/tests/unit/test_memory/test_cascade/test_handlers_daily_log_mapping.py +++ b/tests/unit/test_memory/test_cascade/test_handlers_daily_log_mapping.py @@ -1,4 +1,4 @@ -"""Per-kind ``_build_row`` mapping for the 3 non-Episode daily-log handlers. +"""Per-kind ``_build_row`` mapping for the 4 non-Episode daily-log handlers. The diff loop (read → sha256 → 3-way diff → upsert/delete) lives on :class:`BaseDailyLogHandler` and is exercised by @@ -8,7 +8,7 @@ Each kind gets one happy-path test (all fields present) plus a focused error-path test (missing required inline field). Sharing one -file avoids 3 nearly-identical fixture stacks. +file avoids nearly-identical fixture stacks. """ from __future__ import annotations @@ -20,9 +20,11 @@ from everos.component.embedding import EmbeddingCapability, EmbeddingProvider from everos.component.tokenizer import Tokenizer from everos.core.persistence import MemoryRoot, StructuredEntry +from everos.infra.persistence.lancedb import Decision as LanceDecision from everos.memory.cascade.handlers import ( AgentCaseHandler, AtomicFactHandler, + DecisionHandler, ForesightHandler, HandlerDeps, ) @@ -195,6 +197,179 @@ async def test_foresight_optional_evidence_left_none(tmp_path) -> None: # type: assert row.duration_days is None +# ── Decision ───────────────────────────────────────────────────────────── + + +async def test_decision_build_row_maps_sections_and_tags(tmp_path) -> None: # type: ignore[no-untyped-def] + handler = DecisionHandler(_deps(tmp_path)) + row = await handler._build_row( + owner_id="u1", + owner_type="user", + md_path="users/u1/decisions/decision-2026-05-14.md", + entry=_entry( + "dc_20260514_0001", + inline={ + "owner_id": "u1", + "session_id": "s1", + "timestamp": "2026-05-14T10:00:00+00:00", + "parent_id": "mc_1", + "tags": "[runtime, rust]", + }, + sections={ + "Title": "Use Rust for the device Runtime", + "Decision": "Ship the device Runtime in Rust.", + "Reason": "Need deterministic latency without a GC pause.", + "Impact": "Device builds take longer to compile.", + }, + ), + ) + assert row.id == "u1_dc_20260514_0001" + assert row.title == "Use Rust for the device Runtime" + assert row.decision == "Ship the device Runtime in Rust." + assert row.decision_tokens == "Ship the device Runtime in Rust." + assert row.reason == "Need deterministic latency without a GC pause." + assert row.reason_tokens == "Need deterministic latency without a GC pause." + assert row.impact == "Device builds take longer to compile." + assert row.tags == ["runtime", "rust"] + assert row.parent_id == "mc_1" + assert row.parent_type == "memcell" + assert row.timestamp == _dt.datetime(2026, 5, 14, 10, 0, tzinfo=_dt.UTC) + assert len(row.vector) == 1024 + assert "sender_ids" not in LanceDecision.model_fields + + +async def test_decision_embed_is_fed_decision_body_only( + tmp_path, monkeypatch: pytest.MonkeyPatch +) -> None: # type: ignore[no-untyped-def] + """Vector source is the Decision body, not Title / Reason / Impact.""" + import everos.component.embedding.accessor as acc + + class _RecordingEmbedder(EmbeddingProvider): + dim = 1024 + + def __init__(self) -> None: + self.texts: list[str] = [] + + async def embed(self, text: str) -> list[float]: + self.texts.append(text) + return [0.0] * self.dim + + async def embed_batch(self, texts): # type: ignore[no-untyped-def] + return [await self.embed(t) for t in texts] + + recorder = _RecordingEmbedder() + monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=recorder)) + handler = DecisionHandler(_deps(tmp_path)) + await handler._build_row( + owner_id="u1", + owner_type="user", + md_path="x.md", + entry=_entry( + "dc_20260514_0001", + inline={ + "owner_id": "u1", + "session_id": "s1", + "timestamp": "2026-05-14T10:00:00+00:00", + "parent_id": "mc_1", + }, + sections={ + "Title": "Use Rust for the device Runtime", + "Decision": "Ship the device Runtime in Rust.", + "Reason": "Need deterministic latency without a GC pause.", + "Impact": "Device builds take longer to compile.", + }, + ), + ) + assert recorder.texts == ["Ship the device Runtime in Rust."] + + +async def test_decision_optional_impact_left_none(tmp_path) -> None: # type: ignore[no-untyped-def] + handler = DecisionHandler(_deps(tmp_path)) + row = await handler._build_row( + owner_id="u1", + owner_type="user", + md_path="x.md", + entry=_entry( + "dc_20260514_0001", + inline={ + "owner_id": "u1", + "session_id": "s1", + "timestamp": "2026-05-14T10:00:00+00:00", + "parent_id": "mc_1", + }, + sections={ + "Title": "t", + "Decision": "d", + "Reason": "r", + }, + ), + ) + assert row.impact is None + assert row.reason_tokens == "r" + + +@pytest.mark.parametrize("tags_inline", [None, "[]"]) +async def test_decision_empty_tags_are_empty_list( + tmp_path, tags_inline: str | None +) -> None: # type: ignore[no-untyped-def] + inline = { + "owner_id": "u1", + "session_id": "s1", + "timestamp": "2026-05-14T10:00:00+00:00", + "parent_id": "mc_1", + } + if tags_inline is not None: + inline["tags"] = tags_inline + handler = DecisionHandler(_deps(tmp_path)) + row = await handler._build_row( + owner_id="u1", + owner_type="user", + md_path="x.md", + entry=_entry( + "dc_20260514_0001", + inline=inline, + sections={"Title": "t", "Decision": "d", "Reason": "r"}, + ), + ) + assert row.tags == [] + + +async def test_decision_empty_reason_still_tokenizes(tmp_path) -> None: # type: ignore[no-untyped-def] + handler = DecisionHandler(_deps(tmp_path)) + row = await handler._build_row( + owner_id="u1", + owner_type="user", + md_path="x.md", + entry=_entry( + "dc_20260514_0001", + inline={ + "owner_id": "u1", + "session_id": "s1", + "timestamp": "2026-05-14T10:00:00+00:00", + "parent_id": "mc_1", + }, + sections={"Title": "t", "Decision": "d", "Reason": ""}, + ), + ) + assert row.reason == "" + assert row.reason_tokens == "" + + +async def test_decision_missing_timestamp_raises(tmp_path) -> None: # type: ignore[no-untyped-def] + handler = DecisionHandler(_deps(tmp_path)) + with pytest.raises(ValueError, match="timestamp"): + await handler._build_row( + owner_id="u1", + owner_type="user", + md_path="x.md", + entry=_entry( + "dc_20260514_0001", + inline={"owner_id": "u1", "session_id": "s1", "parent_id": "mc_1"}, + sections={"Title": "t", "Decision": "d", "Reason": "r"}, + ), + ) + + # ── AgentCase ──────────────────────────────────────────────────────────── diff --git a/tests/unit/test_memory/test_cascade/test_handlers_embed_or_none.py b/tests/unit/test_memory/test_cascade/test_handlers_embed_or_none.py index 3e19a3b1d..060c70470 100644 --- a/tests/unit/test_memory/test_cascade/test_handlers_embed_or_none.py +++ b/tests/unit/test_memory/test_cascade/test_handlers_embed_or_none.py @@ -24,6 +24,7 @@ AgentCaseHandler, AgentSkillHandler, AtomicFactHandler, + DecisionHandler, ForesightHandler, HandlerDeps, ) @@ -129,6 +130,34 @@ async def test_foresight_handler_writes_null_vector_when_embed_unavailable( assert row.foresight_tokens == "user will book lunch" +async def test_decision_handler_writes_null_vector_when_embed_unavailable( + memory_root: MemoryRoot, +) -> None: + handler = DecisionHandler(_deps(memory_root)) + row = await handler._build_row( + owner_id="u1", + owner_type="user", + md_path="x.md", + entry=_entry( + "dc_20260514_0001", + inline={ + "owner_id": "u1", + "session_id": "s1", + "timestamp": "2026-05-14T10:00:00+00:00", + "parent_id": "mc_1", + }, + sections={ + "Title": "Use Rust for the device Runtime", + "Decision": "Ship the device Runtime in Rust.", + "Reason": "Need deterministic latency without a GC pause.", + }, + ), + ) + assert row.vector is None + assert row.decision_tokens == "Ship the device Runtime in Rust." + assert row.reason_tokens == "Need deterministic latency without a GC pause." + + async def test_agent_case_handler_writes_null_vector_when_embed_unavailable( memory_root: MemoryRoot, ) -> None: diff --git a/tests/unit/test_memory/test_cascade/test_registry.py b/tests/unit/test_memory/test_cascade/test_registry.py index 8db5b2696..75c3bba10 100644 --- a/tests/unit/test_memory/test_cascade/test_registry.py +++ b/tests/unit/test_memory/test_cascade/test_registry.py @@ -1,6 +1,6 @@ """Tests for the cascade kind registry. -Verify the 5 registered kinds' globs match the right paths and reject +Verify the registered kinds' globs match the right paths and reject noise (random ``.md``, swp files, profile-style paths). ``match_kind`` must walk the registry in declared order and pick the first matching spec. @@ -29,6 +29,14 @@ "default_app/default_project/users/u1/.foresights/foresight-2026-05-14.md", "foresight", ), + ( + "default_app/default_project/users/u1/decisions/decision-2026-05-14.md", + "decision", + ), + ( + "default_app/default_project/users/u1/principles.md", + "principle", + ), ( "default_app/default_project/agents/a1/.cases/agent_case-2026-05-14.md", "agent_case", @@ -58,22 +66,33 @@ def test_match_kind_recognises_registered_paths(path: str, expected_kind: str) - # rejected so a prefix-less path can never silently match (the scanner # would otherwise find nothing while the watcher matched, a split brain). "users/u1/episodes/episode-2026-05-14.md", + "users/u1/decisions/decision-2026-05-14.md", + "users/u1/principles.md", + # Dot-prefixed decisions dir is not the product path. + "users/u1/.decisions/decision-2026-05-14.md", + "default_app/default_project/users/u1/.decisions/decision-2026-05-14.md", ], ) def test_match_kind_rejects_unregistered_paths(path: str) -> None: assert match_kind(path) is None -def test_registry_has_exactly_eight_kinds() -> None: - """The registry pins the cascade surface — no silent registration.""" +def test_registry_has_exactly_ten_kinds() -> None: + """The registry pins the cascade surface — no silent registration. + + ``principle`` is a cascade projection name (Meta Memory), not a + product Kind; it still occupies one KindSpec so md can reach Lance. + """ names = [s.name for s in KIND_REGISTRY] assert names == [ "episode", "atomic_fact", "foresight", + "decision", "agent_case", "agent_skill", "user_profile", + "principle", "knowledge_document", "knowledge_topic", ] diff --git a/tests/unit/test_memory/test_cascade/test_registry_knowledge_gate.py b/tests/unit/test_memory/test_cascade/test_registry_knowledge_gate.py index 250460ea7..7ed15b866 100644 --- a/tests/unit/test_memory/test_cascade/test_registry_knowledge_gate.py +++ b/tests/unit/test_memory/test_cascade/test_registry_knowledge_gate.py @@ -149,8 +149,10 @@ def test_non_knowledge_handlers_always_registered( "episode", "atomic_fact", "foresight", + "decision", "agent_case", "agent_skill", "user_profile", + "principle", ): assert kind in handlers diff --git a/tests/unit/test_memory/test_cascade/test_scanner_kinds.py b/tests/unit/test_memory/test_cascade/test_scanner_kinds.py index b12b23064..b89879962 100644 --- a/tests/unit/test_memory/test_cascade/test_scanner_kinds.py +++ b/tests/unit/test_memory/test_cascade/test_scanner_kinds.py @@ -28,6 +28,24 @@ def _make_episode_md(root: Path) -> Path: return p +def _make_decision_md(root: Path) -> Path: + """Create a plausible decision daily-log file under the memory root.""" + d = root / "default_app" / "default_project" / "users" / "u1" / "decisions" + d.mkdir(parents=True, exist_ok=True) + p = d / "decision-2026-01-01.md" + p.write_text("ok") + return p + + +def _make_principles_md(root: Path) -> Path: + """Create a plausible principles.md under the memory root.""" + d = root / "default_app" / "default_project" / "users" / "u1" + d.mkdir(parents=True, exist_ok=True) + p = d / "principles.md" + p.write_text("ok") + return p + + def _make_agent_skill_md(root: Path) -> Path: """Create a plausible SKILL.md under the memory root.""" d = ( @@ -47,14 +65,18 @@ def _make_agent_skill_md(root: Path) -> Path: def test_collect_scan_inputs_default_scans_all_kinds(tmp_path: Path) -> None: """Default (``kinds=None``) walks every :data:`KIND_REGISTRY` path - glob — episode + skill both surface.""" + glob — episode + skill + decision all surface.""" _make_episode_md(tmp_path) _make_agent_skill_md(tmp_path) + _make_decision_md(tmp_path) + _make_principles_md(tmp_path) inputs = _collect_scan_inputs(tmp_path) kinds = {i.kind for i in inputs} assert "episode" in kinds assert "agent_skill" in kinds + assert "decision" in kinds + assert "principle" in kinds def test_collect_scan_inputs_with_kinds_filter_restricts_to_named( diff --git a/tests/unit/test_memory/test_events.py b/tests/unit/test_memory/test_events.py index 1328bcc09..c8f397559 100644 --- a/tests/unit/test_memory/test_events.py +++ b/tests/unit/test_memory/test_events.py @@ -7,6 +7,8 @@ from everos.memory.events import ( AgentCaseExtracted, AgentPipelineStarted, + DecisionClusterUpdated, + DecisionExtracted, SkillClusterUpdated, UserPipelineStarted, ) @@ -129,3 +131,85 @@ def test_skill_cluster_updated_carries_case_vector() -> None: SkillClusterUpdated.model_validate(payload).model_dump_json() ) assert event.case_vector == [0.1, 0.2, 0.3] + + +def test_decision_extracted_topic_is_module_qualified() -> None: + assert DecisionExtracted.topic() == "everos.memory.events:DecisionExtracted" + + +def test_decision_extracted_roundtrip_json() -> None: + event = DecisionExtracted( + memcell_id="mc_a", + decision_entry_id="dc_20260517_00000001", + title="Use Rust on device", + decision_text="Device Runtime uses Rust.", + reason="Need deterministic latency.", + impact="Keep Python in the agent runtime.", + tags=["runtime"], + decision_timestamp_ms=1_700_000_000_000, + owner_id="u_alice", + session_id="s1", + ) + restored = DecisionExtracted.model_validate_json(event.model_dump_json()) + assert restored.decision_entry_id == "dc_20260517_00000001" + assert restored.decision_text == "Device Runtime uses Rust." + assert restored.tags == ["runtime"] + assert restored.impact == "Keep Python in the agent runtime." + assert restored.source == "pipeline" + + +def test_decision_extracted_source_defaults_to_pipeline() -> None: + event = DecisionExtracted( + memcell_id="mc_a", + decision_entry_id="dc_1", + title="T", + decision_text="D", + reason="R", + decision_timestamp_ms=1, + owner_id="u_alice", + ) + assert event.source == "pipeline" + assert event.impact is None + assert event.tags == [] + assert event.session_id is None + + +def test_decision_cluster_updated_topic_is_module_qualified() -> None: + assert DecisionClusterUpdated.topic() == ( + "everos.memory.events:DecisionClusterUpdated" + ) + + +def test_decision_cluster_updated_roundtrip_json() -> None: + event = DecisionClusterUpdated( + memcell_id="mc_a", + decision_entry_id="dc_20260517_00000001", + cluster_id="cl_dec00000001", + owner_id="u_alice", + title="Use Rust on device", + decision_text="Device Runtime uses Rust.", + reason="Need deterministic latency.", + impact="Keep Python in the agent runtime.", + tags=["runtime"], + decision_timestamp_ms=1_700_000_000_000, + ) + restored = DecisionClusterUpdated.model_validate_json(event.model_dump_json()) + assert restored.decision_entry_id == "dc_20260517_00000001" + assert restored.cluster_id == "cl_dec00000001" + assert restored.decision_text == "Device Runtime uses Rust." + assert restored.tags == ["runtime"] + + +def test_decision_cluster_updated_snapshot_defaults_for_back_compat() -> None: + event = DecisionClusterUpdated( + memcell_id="mc_a", + decision_entry_id="dc_1", + cluster_id="cl_1", + owner_id="u_alice", + ) + assert event.title == "" + assert event.decision_text == "" + assert event.reason == "" + assert event.impact is None + assert event.tags == [] + assert event.decision_timestamp_ms == 0 diff --git a/tests/unit/test_memory/test_extract/test_pipeline/test_user_memory_emits.py b/tests/unit/test_memory/test_extract/test_pipeline/test_user_memory_emits.py index f696d7244..eeb802b4c 100644 --- a/tests/unit/test_memory/test_extract/test_pipeline/test_user_memory_emits.py +++ b/tests/unit/test_memory/test_extract/test_pipeline/test_user_memory_emits.py @@ -99,7 +99,10 @@ async def test_emit_episode_extracted_after_md_write() -> None: ], ) algo_ep = AlgoEpisode( - owner_id="u1", episode="they said hello", timestamp=1_700_000_000_000 + owner_id="u1", + episode="they said hello", + summary="they said hello", + timestamp=1_700_000_000_000, ) with patch.object( pipeline._ep_ext, "aextract", new=AsyncMock(return_value=algo_ep) @@ -167,7 +170,10 @@ async def test_run_emits_extract_and_persist_spans() -> None: ], ) algo_ep = AlgoEpisode( - owner_id="u1", episode="they said hello", timestamp=1_700_000_000_000 + owner_id="u1", + episode="they said hello", + summary="they said hello", + timestamp=1_700_000_000_000, ) exporter = InMemorySpanExporter() @@ -241,7 +247,10 @@ async def test_extract_persist_capture_content_when_on() -> None: ], ) algo_ep = AlgoEpisode( - owner_id="u1", episode="they said hello", timestamp=1_700_000_000_000 + owner_id="u1", + episode="they said hello", + summary="they said hello", + timestamp=1_700_000_000_000, ) exporter = InMemorySpanExporter() diff --git a/tests/unit/test_memory/test_get/test_dto.py b/tests/unit/test_memory/test_get/test_dto.py index e9b81cf95..2eae04e6e 100644 --- a/tests/unit/test_memory/test_get/test_dto.py +++ b/tests/unit/test_memory/test_get/test_dto.py @@ -18,6 +18,7 @@ from pydantic import ValidationError from everos.memory.get.dto import ( + GetData, GetMemoryType, GetRequest, ) @@ -114,7 +115,7 @@ def test_get_request_rejects_both_user_and_agent_id() -> None: def test_get_request_rejects_invalid_memory_type_value() -> None: - """A value outside the four-kind enum is 422.""" + """A value outside the enumerated kinds is 422.""" with pytest.raises(ValidationError): GetRequest.model_validate( { @@ -124,6 +125,26 @@ def test_get_request_rejects_invalid_memory_type_value() -> None: ) +def test_get_request_rejects_principle_memory_type() -> None: + """Principle is Meta Memory — not a GetMemoryType.""" + with pytest.raises(ValidationError): + GetRequest.model_validate( + { + "user_id": "u1", + "memory_type": "principle", + } + ) + + +def test_get_data_defaults_include_decisions() -> None: + data = GetData() + assert data.episodes == [] + assert data.decisions == [] + assert data.profiles == [] + assert data.agent_cases == [] + assert data.agent_skills == [] + + def test_get_request_rejects_invalid_sort_order() -> None: """``sort_order`` is a tight Literal — typos / casing variants are 422.""" with pytest.raises(ValidationError): @@ -143,6 +164,7 @@ def test_get_request_rejects_invalid_sort_order() -> None: "id_field, memory_type", [ ("user_id", GetMemoryType.EPISODE), + ("user_id", GetMemoryType.DECISION), ("user_id", GetMemoryType.PROFILE), ("agent_id", GetMemoryType.AGENT_CASE), ("agent_id", GetMemoryType.AGENT_SKILL), @@ -152,7 +174,7 @@ def test_get_request_allows_valid_owner_memory_pair( id_field: str, memory_type: GetMemoryType, ) -> None: - """The four valid (owner-kind, memory_type) combinations.""" + """The valid (owner-kind, memory_type) combinations.""" req = GetRequest(**{id_field: "u1"}, memory_type=memory_type) assert req.memory_type is memory_type expected_owner_type = "user" if id_field == "user_id" else "agent" @@ -165,6 +187,7 @@ def test_get_request_allows_valid_owner_memory_pair( ("user_id", GetMemoryType.AGENT_CASE), ("user_id", GetMemoryType.AGENT_SKILL), ("agent_id", GetMemoryType.EPISODE), + ("agent_id", GetMemoryType.DECISION), ("agent_id", GetMemoryType.PROFILE), ], ) diff --git a/tests/unit/test_memory/test_get/test_manager.py b/tests/unit/test_memory/test_get/test_manager.py index 35cec6b79..1496eebf4 100644 --- a/tests/unit/test_memory/test_get/test_manager.py +++ b/tests/unit/test_memory/test_get/test_manager.py @@ -24,6 +24,7 @@ from everos.infra.persistence.lancedb import ( AgentCase, AgentSkill, + Decision, Episode, UserProfile, ) @@ -114,6 +115,29 @@ def _episode_row(entry: str) -> Episode: ) +def _decision_row(entry: str) -> Decision: + return Decision( + id=f"u1_{entry}", + entry_id=entry, + owner_id="u1", + owner_type="user", + session_id="sess_a", + timestamp=_ts(), + parent_type="memcell", + parent_id="mc_1", + title=f"title {entry}", + decision=f"decision {entry}", + reason=f"reason {entry}", + impact=None, + tags=["runtime"], + decision_tokens=f"decision {entry}", + reason_tokens=f"reason {entry}", + md_path=f"users/u1/decisions/{entry}.md", + content_sha256="abc", + vector=[0.0] * 1024, + ) + + def _agent_case_row(entry: str) -> AgentCase: return AgentCase( id=f"a1_{entry}", @@ -179,26 +203,28 @@ def profile_repo() -> _ProfileStubRepo: @pytest.fixture def manager( profile_repo: _ProfileStubRepo, -) -> tuple[GetManager, _StubRepo, _StubRepo, _StubRepo]: +) -> tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo]: ep = _StubRepo() + dc = _StubRepo() ac = _StubRepo() sk = _StubRepo() mgr = GetManager( episode_repo=ep, # type: ignore[arg-type] + decision_repo=dc, # type: ignore[arg-type] agent_case_repo=ac, # type: ignore[arg-type] agent_skill_repo=sk, # type: ignore[arg-type] user_profile_repo=profile_repo, # type: ignore[arg-type] ) - return mgr, ep, ac, sk + return mgr, ep, dc, ac, sk # ── Episode dispatch ──────────────────────────────────────────────────── async def test_episodic_memory_populates_episodes_and_counts( - manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo], + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], ) -> None: - mgr, ep, _, _ = manager + mgr, ep, *_ = manager ep.rows = [_episode_row("ep_1"), _episode_row("ep_2")] ep.total = 17 # filtered total may exceed the page req = GetRequest( @@ -214,6 +240,7 @@ async def test_episodic_memory_populates_episodes_and_counts( assert resp.data.count == 2 assert [item.id for item in resp.data.episodes] == ["u1_ep_1", "u1_ep_2"] assert resp.data.profiles == [] + assert resp.data.decisions == [] assert resp.data.agent_cases == [] assert resp.data.agent_skills == [] # The shaper maps the lance row's owner_id onto the item's user_id field. @@ -221,13 +248,13 @@ async def test_episodic_memory_populates_episodes_and_counts( async def test_get_uses_propagated_request_id_when_bound( - manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo], + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], ) -> None: """When a request id is bound upstream (middleware), ``get`` reuses it instead of minting a fresh one, so the response id matches the trace.""" from everos.core.context import reset_request_id, set_request_id - mgr, ep, _, _ = manager + mgr, ep, *_ = manager ep.rows = [_episode_row("ep_1")] token = set_request_id("deadbeef" * 4) try: @@ -240,10 +267,10 @@ async def test_get_uses_propagated_request_id_when_bound( async def test_episodic_memory_passes_where_and_sort_to_repo( - manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo], + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], ) -> None: """The compiled ``where`` must include owner_id + filter clauses.""" - mgr, ep, _, _ = manager + mgr, ep, *_ = manager req = GetRequest( user_id="u1", memory_type=GetMemoryType.EPISODE, @@ -263,14 +290,59 @@ async def test_episodic_memory_passes_where_and_sort_to_repo( assert ep.last.page_size == 10 +# ── Decision dispatch ─────────────────────────────────────────────────── + + +async def test_decision_memory_populates_decisions_and_counts( + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], +) -> None: + mgr, _, dc, *_ = manager + dc.rows = [_decision_row("dc_1"), _decision_row("dc_2")] + dc.total = 4 + req = GetRequest(user_id="u1", memory_type=GetMemoryType.DECISION) + resp = await mgr.get(req) + + assert resp.data.total_count == 4 + assert resp.data.count == 2 + assert [item.id for item in resp.data.decisions] == ["u1_dc_1", "u1_dc_2"] + assert resp.data.episodes == [] + assert resp.data.profiles == [] + item = resp.data.decisions[0] + assert item.user_id == "u1" + assert item.title == "title dc_1" + assert item.decision == "decision dc_1" + assert item.reason == "reason dc_1" + assert item.tags == ["runtime"] + assert item.impact is None + assert item.session_id == "sess_a" + + +async def test_decision_where_excludes_deprecated_and_strips_sender_id( + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], +) -> None: + mgr, ep, dc, *_ = manager + req = GetRequest( + user_id="u1", + memory_type=GetMemoryType.DECISION, + filters=FilterNode.model_validate({"sender_id": "u1", "session_id": "sess_a"}), + ) + await mgr.get(req) + assert "deprecated_by IS NULL" in dc.last.where + assert "sender_ids" not in dc.last.where + assert "session_id = 'sess_a'" in dc.last.where + assert dc.last.sort_by == "timestamp" + # Episode repo is not touched on the decision path. + assert ep.last.where == "" + + # ── Profile dispatch ──────────────────────────────────────────────────── async def test_profile_miss_returns_empty( - manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo], + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], ) -> None: """Cold start (no profile row yet) → empty list + total_count=0.""" - mgr, ep, ac, sk = manager # profile_repo.row defaults to None + mgr, ep, dc, ac, sk = manager # profile_repo.row defaults to None req = GetRequest( user_id="u1", memory_type=GetMemoryType.PROFILE, @@ -281,12 +353,13 @@ async def test_profile_miss_returns_empty( assert resp.data.count == 0 # The profile path never touches the paginated (episode/case/skill) repos. assert ep.last.where == "" + assert dc.last.where == "" assert ac.last.where == "" assert sk.last.where == "" async def test_profile_hit_shapes_row_into_item( - manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo], + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], profile_repo: _ProfileStubRepo, ) -> None: """A present profile row is fetched by owner and shaped + json-decoded.""" @@ -316,9 +389,9 @@ async def test_profile_hit_shapes_row_into_item( async def test_agent_case_populates_agent_cases( - manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo], + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], ) -> None: - mgr, _, ac, _ = manager + mgr, _, _, ac, _ = manager ac.rows = [_agent_case_row("ac_1"), _agent_case_row("ac_2")] ac.total = 2 req = GetRequest( @@ -330,6 +403,7 @@ async def test_agent_case_populates_agent_cases( assert resp.data.count == 2 assert [item.id for item in resp.data.agent_cases] == ["a1_ac_1", "a1_ac_2"] assert resp.data.episodes == [] + assert resp.data.decisions == [] assert resp.data.agent_skills == [] @@ -337,10 +411,10 @@ async def test_agent_case_populates_agent_cases( async def test_agent_skill_sort_by_silently_overridden_to_updated_at( - manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo], + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], ) -> None: """``agent_skill`` always sorts by ``updated_at`` (no ``timestamp`` column).""" - mgr, _, _, sk = manager + mgr, *_, sk = manager sk.rows = [_agent_skill_row("planner")] sk.total = 1 req = GetRequest( @@ -356,10 +430,10 @@ async def test_agent_skill_sort_by_silently_overridden_to_updated_at( async def test_agent_skill_explicit_updated_at_is_respected( - manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo], + manager: tuple[GetManager, _StubRepo, _StubRepo, _StubRepo, _StubRepo], ) -> None: """``updated_at`` passes through unchanged (no double-override surprise).""" - mgr, _, _, sk = manager + mgr, *_, sk = manager req = GetRequest( agent_id="a1", memory_type=GetMemoryType.AGENT_SKILL, diff --git a/tests/unit/test_memory/test_models.py b/tests/unit/test_memory/test_models.py index 91f6c4839..a3b01000a 100644 --- a/tests/unit/test_memory/test_models.py +++ b/tests/unit/test_memory/test_models.py @@ -16,6 +16,9 @@ from everalgo.types import ( AtomicFact as AlgoAtomicFact, ) +from everalgo.types import ( + Decision as AlgoDecision, +) from everalgo.types import ( Episode as AlgoEpisode, ) @@ -23,7 +26,7 @@ Foresight as AlgoForesight, ) -from everos.memory.models import AgentCase, AtomicFact, Episode, Foresight +from everos.memory.models import AgentCase, AtomicFact, Decision, Episode, Foresight def test_atomic_fact_from_algo_carries_business_fields_and_metadata() -> None: @@ -174,7 +177,9 @@ def test_episode_from_algo_owner_id_caller_supplied() -> None: EPISODE_GENERATION_PROMPT) and then fans the same algo Episode out to one domain Episode per user sender, each rooted at its own owner. """ - algo = AlgoEpisode(owner_id=None, episode="hello", timestamp=1_700_000_000_000) + algo = AlgoEpisode( + owner_id=None, episode="hello", summary="hello", timestamp=1_700_000_000_000 + ) ep_alice = Episode.from_algo( algo, owner_id="u_alice", @@ -194,3 +199,57 @@ def test_episode_from_algo_owner_id_caller_supplied() -> None: assert ep_alice.episode == ep_bob.episode == "hello" assert ep_alice.parent_id == ep_bob.parent_id == "mc_a" assert ep_alice.session_id == ep_bob.session_id == "s1" + + +def _algo_decision(**overrides: object) -> AlgoDecision: + item: dict[str, object] = { + "owner_id": None, + "title": "Agent Runtime language", + "decision": "Python on the core, Rust on device.", + "reason": "Iterate the agent surface; keep the device loop stable.", + "impact": "Device capabilities connect through APIs.", + "tags": ["architecture", "runtime"], + "timestamp": 1_700_000_000_000, + } + item.update(overrides) + return AlgoDecision.model_validate(item) + + +def test_decision_from_algo_owner_id_caller_supplied() -> None: + """Caller supplies ``owner_id``; algo's value (None or otherwise) is dropped. + + DecisionExtractor runs once per MemCell with no ``sender_id``. EverOS then + fans the same algo Decision out to one domain Decision per user sender. + """ + algo = _algo_decision(owner_id=None) + dc_alice = Decision.from_algo( + algo, owner_id="u_alice", session_id="s1", parent_id="mc_a" + ) + dc_bob = Decision.from_algo( + algo, owner_id="u_bob", session_id="s1", parent_id="mc_a" + ) + assert dc_alice.owner_id == "u_alice" + assert dc_bob.owner_id == "u_bob" + assert dc_alice.decision == dc_bob.decision == algo.decision + assert dc_alice.title == dc_bob.title == algo.title + assert dc_alice.reason == dc_bob.reason == algo.reason + assert dc_alice.impact == algo.impact + assert dc_alice.tags == algo.tags + assert dc_alice.parent_id == dc_bob.parent_id == "mc_a" + assert dc_alice.session_id == dc_bob.session_id == "s1" + assert not hasattr(dc_alice, "sender_ids") + + +def test_decision_from_algo_overrides_stale_algo_owner_id() -> None: + """Fan-out still wins when the LLM smuggles a non-None owner_id.""" + algo = _algo_decision(owner_id="PLACEHOLDER") + dc = Decision.from_algo(algo, owner_id="u_alice", session_id="s1", parent_id="mc_a") + assert dc.owner_id == "u_alice" + + +def test_decision_from_algo_drops_algo_side_parent_id() -> None: + algo = _algo_decision(parent_id="ALGO_STALE") + dc = Decision.from_algo( + algo, owner_id="u_alice", session_id="s1", parent_id="mc_real" + ) + assert dc.parent_id == "mc_real" diff --git a/tests/unit/test_memory/test_reflection/test_decision_orchestrator.py b/tests/unit/test_memory/test_reflection/test_decision_orchestrator.py new file mode 100644 index 000000000..ec0327448 --- /dev/null +++ b/tests/unit/test_memory/test_reflection/test_decision_orchestrator.py @@ -0,0 +1,427 @@ +"""Tests for :class:`DecisionReflectionOrchestrator`. + +Constructor dependencies are mocked. Tests verify: +- candidate selection filtering (INIT vs UPDATE) with ``kind=decision`` +- full INIT-mode flow with merge + deprecate (no atomic facts) +- UPDATE-mode ``old_decision`` passthrough +- LLM failure skips cluster gracefully +- empty candidates return empty list +""" + +from __future__ import annotations + +import datetime as _dt +from dataclasses import dataclass, field +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from everos.infra.ome.testing import FakeStrategyContext +from everos.memory._partition_locks import _reset_for_tests +from everos.memory.events import DecisionExtracted, EpisodeExtracted +from everos.memory.reflection.decision_orchestrator import ( + _MAX_CLUSTERS_PER_RUN, + DecisionReflectionOrchestrator, + _merged_decision_to_entry_body, + _ts_to_ms, +) + + +@pytest.fixture(autouse=True) +def _isolate_locks() -> None: + _reset_for_tests() + + +@dataclass +class _FakeAlgoResult: + """Minimal stand-in for ``everalgo.types.Decision``.""" + + owner_id: str | None + title: str + decision: str + reason: str + timestamp: int + impact: str | None = None + tags: list[str] = field(default_factory=list) + + +@dataclass +class _FakeDecisionRow: + """Minimal stand-in for a LanceDB Decision row.""" + + id: str + entry_id: str + owner_id: str + owner_type: str = "user" + app_id: str = "default" + project_id: str = "default" + session_id: str | None = "s_test" + timestamp: _dt.datetime = _dt.datetime(2026, 6, 1, tzinfo=_dt.UTC) + parent_type: str = "memcell" + parent_id: str = "mc_aaa" + title: str = "test title" + decision: str = "test decision text" + reason: str = "test reason" + impact: str | None = None + tags: list[str] = field(default_factory=lambda: ["runtime"]) + md_path: str = "users/u_alice/decisions/decision-2026-06-01.md" + content_sha256: str = "abc123" + deprecated_by: str | None = None + vector: list[float] | None = None + + +def _make_decision_row( + entry_id: str = "dc_20260601_0001", + parent_id: str = "mc_aaa", + parent_type: str = "memcell", + owner_id: str = "u_alice", + **kwargs: object, +) -> _FakeDecisionRow: + return _FakeDecisionRow( + id=f"{owner_id}_{entry_id}", + entry_id=entry_id, + owner_id=owner_id, + parent_id=parent_id, + parent_type=parent_type, + **kwargs, # type: ignore[arg-type] + ) + + +def _make_entry_id(formatted: str = "dc_20260614_0001") -> MagicMock: + eid = MagicMock() + eid.format.return_value = formatted + eid.date = _dt.date(2026, 6, 14) + return eid + + +def _build_orchestrator( + *, + cluster_repo: MagicMock | None = None, + decision_store: MagicMock | None = None, + decision_writer: MagicMock | None = None, + report_repo: MagicMock | None = None, + reflector: MagicMock | None = None, + embedder: MagicMock | None = None, +) -> DecisionReflectionOrchestrator: + return DecisionReflectionOrchestrator( + cluster_repo=cluster_repo or MagicMock(), + decision_store=decision_store or MagicMock(), + decision_writer=decision_writer or MagicMock(), + report_repo=report_repo or MagicMock(), + reflector=reflector or MagicMock(), + embedder=embedder or MagicMock(), + ) + + +async def test_select_candidates_init_and_update() -> None: + """Unreflected clusters with >=2 members are INIT candidates. + Reflected clusters with >1 member are UPDATE candidates. + """ + cluster_repo = MagicMock() + report_repo = MagicMock() + + report_repo.list_reflected_cluster_ids = AsyncMock(return_value={"cl_reflected"}) + cluster_repo.list_ids_and_member_counts = AsyncMock( + return_value=[ + ("cl_new_3", 3), + ("cl_new_1", 1), + ("cl_reflected", 2), + ("cl_reflected_1", 1), + ] + ) + + orch = _build_orchestrator(cluster_repo=cluster_repo, report_repo=report_repo) + result = await orch._select_candidates( + owner_id="u_alice", + kind="decision", + app_id="default", + project_id="default", + ) + + assert result == ["cl_new_3", "cl_reflected"] + cluster_repo.list_ids_and_member_counts.assert_awaited_once_with( + "u_alice", "decision", app_id="default", project_id="default" + ) + + +async def test_select_candidates_respects_max_limit() -> None: + """More than ``_MAX_CLUSTERS_PER_RUN`` candidates are truncated.""" + cluster_repo = MagicMock() + report_repo = MagicMock() + report_repo.list_reflected_cluster_ids = AsyncMock(return_value=set()) + cluster_repo.list_ids_and_member_counts = AsyncMock( + return_value=[(f"cl_{i:03d}", i + 2) for i in range(_MAX_CLUSTERS_PER_RUN + 5)] + ) + + orch = _build_orchestrator(cluster_repo=cluster_repo, report_repo=report_repo) + result = await orch._select_candidates( + owner_id="u_alice", + kind="decision", + app_id="default", + project_id="default", + ) + assert len(result) == _MAX_CLUSTERS_PER_RUN + + +async def test_empty_candidates_returns_empty() -> None: + """No qualifying clusters -> run() returns empty list immediately.""" + cluster_repo = MagicMock() + report_repo = MagicMock() + report_repo.list_reflected_cluster_ids = AsyncMock(return_value=set()) + cluster_repo.list_ids_and_member_counts = AsyncMock( + return_value=[("cl_only_one", 1)] + ) + + orch = _build_orchestrator(cluster_repo=cluster_repo, report_repo=report_repo) + ctx = FakeStrategyContext() + reports = await orch.run(ctx=ctx, owner_id="u_alice") + assert reports == [] + cluster_repo.list_ids_and_member_counts.assert_awaited_once_with( + "u_alice", "decision", app_id="default", project_id="default" + ) + + +async def test_run_init_mode_merges_and_deprecates() -> None: + """Full INIT flow: 2 decision members -> merge -> write -> deprecate.""" + cluster_repo = MagicMock() + decision_store = MagicMock() + decision_writer = MagicMock() + report_repo = MagicMock() + reflector = MagicMock() + embedder = MagicMock() + + report_repo.list_reflected_cluster_ids = AsyncMock(return_value=set()) + cluster_repo.list_ids_and_member_counts = AsyncMock(return_value=[("cl_abc", 2)]) + decision_store.find_where = AsyncMock(return_value=[]) + cluster_repo.get_members_with_type = AsyncMock( + return_value=[ + ("dc_20260601_0001", "decision"), + ("dc_20260601_0002", "decision"), + ] + ) + + dc1 = _make_decision_row( + entry_id="dc_20260601_0001", parent_id="mc_001", owner_id="u_alice" + ) + dc2 = _make_decision_row( + entry_id="dc_20260601_0002", parent_id="mc_002", owner_id="u_alice" + ) + decision_store.find_by_owner_entries = AsyncMock(return_value=[dc1, dc2]) + + algo_result = _FakeAlgoResult( + owner_id=None, + title="merged title", + decision="merged decision text", + reason="merged reason", + impact="merged impact", + tags=["runtime"], + timestamp=1717200000000, + ) + reflector.areflect = AsyncMock(return_value=algo_result) + + entry_id_mock = _make_entry_id("dc_20260614_0001") + decision_writer.append_entries = AsyncMock(return_value=[entry_id_mock]) + decision_writer.patch_frontmatter = AsyncMock() + + ctx = FakeStrategyContext() + ctx.wait_for_event = AsyncMock() # type: ignore[method-assign] + + cluster_repo.remove_members = AsyncMock() + cluster_repo.add_member = AsyncMock() + cluster_repo.update_metadata = AsyncMock() + embedder.embed = AsyncMock(return_value=[0.1] * 1024) + decision_store.update = AsyncMock() + report_repo.create = AsyncMock() + + orch = _build_orchestrator( + cluster_repo=cluster_repo, + decision_store=decision_store, + decision_writer=decision_writer, + report_repo=report_repo, + reflector=reflector, + embedder=embedder, + ) + + reports = await orch.run(ctx=ctx, owner_id="u_alice") + + reflector.areflect.assert_awaited_once() + call_kwargs = reflector.areflect.call_args + assert "old_decision" not in (call_kwargs.kwargs or {}) + + decision_writer.append_entries.assert_awaited_once() + ctx.wait_for_event.assert_not_awaited() + + assert len(ctx.emitted) == 1 + event = ctx.emitted[0] + assert isinstance(event, DecisionExtracted) + assert not isinstance(event, EpisodeExtracted) + assert event.source == "reflection" + assert event.session_id is None + assert event.decision_entry_id == "dc_20260614_0001" + assert event.decision_text == "merged decision text" + + cluster_repo.remove_members.assert_awaited_once() + cluster_repo.add_member.assert_awaited_once_with( + "cl_abc", "dc_20260614_0001", "decision" + ) + embedder.embed.assert_awaited_once_with("merged decision text") + decision_store.update.assert_awaited() + + report_repo.create.assert_awaited_once() + created = report_repo.create.await_args.args[0] + assert created.deprecated_fact_count == 0 + assert len(reports) == 1 + + +async def test_run_update_mode_uses_old_decision() -> None: + """UPDATE flow: cluster has 1 merged decision + 1 original decision.""" + cluster_repo = MagicMock() + decision_store = MagicMock() + decision_writer = MagicMock() + report_repo = MagicMock() + reflector = MagicMock() + embedder = MagicMock() + + report_repo.list_reflected_cluster_ids = AsyncMock(return_value={"cl_update"}) + cluster_repo.list_ids_and_member_counts = AsyncMock(return_value=[("cl_update", 2)]) + decision_store.find_where = AsyncMock(return_value=[]) + cluster_repo.get_members_with_type = AsyncMock( + return_value=[ + ("dc_20260612_0001", "decision"), + ("dc_20260613_0001", "decision"), + ] + ) + + old_merged = _make_decision_row( + entry_id="dc_20260612_0001", + parent_id="cl_update", + parent_type="cluster", + owner_id="u_alice", + decision="old merged text", + ) + new_dc = _make_decision_row( + entry_id="dc_20260613_0001", + parent_id="mc_004", + owner_id="u_alice", + decision="new decision text", + ) + decision_store.find_by_owner_entries = AsyncMock(return_value=[new_dc, old_merged]) + + algo_result = _FakeAlgoResult( + owner_id=None, + title="updated title", + decision="updated merged text", + reason="updated reason", + timestamp=1717200000000, + ) + reflector.areflect = AsyncMock(return_value=algo_result) + + entry_id_mock = _make_entry_id("dc_20260614_0002") + decision_writer.append_entries = AsyncMock(return_value=[entry_id_mock]) + decision_writer.patch_frontmatter = AsyncMock() + + cluster_repo.remove_members = AsyncMock() + cluster_repo.add_member = AsyncMock() + cluster_repo.update_metadata = AsyncMock() + embedder.embed = AsyncMock(return_value=[0.1] * 1024) + decision_store.update = AsyncMock() + report_repo.create = AsyncMock() + + ctx = FakeStrategyContext() + orch = _build_orchestrator( + cluster_repo=cluster_repo, + decision_store=decision_store, + decision_writer=decision_writer, + report_repo=report_repo, + reflector=reflector, + embedder=embedder, + ) + + reports = await orch.run(ctx=ctx, owner_id="u_alice") + + reflector.areflect.assert_awaited_once() + _, kwargs = reflector.areflect.call_args + assert "old_decision" in kwargs + assert len(reports) == 1 + + +async def test_llm_failure_skips_cluster() -> None: + """Reflector raising an exception skips the cluster, continues.""" + cluster_repo = MagicMock() + decision_store = MagicMock() + report_repo = MagicMock() + reflector = MagicMock() + + report_repo.list_reflected_cluster_ids = AsyncMock(return_value=set()) + cluster_repo.list_ids_and_member_counts = AsyncMock(return_value=[("cl_fail", 2)]) + decision_store.find_where = AsyncMock(return_value=[]) + cluster_repo.get_members_with_type = AsyncMock( + return_value=[("dc_001", "decision"), ("dc_002", "decision")] + ) + + dc1 = _make_decision_row(entry_id="dc_001", parent_id="mc_a", owner_id="u_alice") + dc2 = _make_decision_row(entry_id="dc_002", parent_id="mc_b", owner_id="u_alice") + decision_store.find_by_owner_entries = AsyncMock(return_value=[dc1, dc2]) + reflector.areflect = AsyncMock(side_effect=RuntimeError("LLM timeout")) + + ctx = FakeStrategyContext() + orch = _build_orchestrator( + cluster_repo=cluster_repo, + decision_store=decision_store, + report_repo=report_repo, + reflector=reflector, + ) + + reports = await orch.run(ctx=ctx, owner_id="u_alice") + assert reports == [] + assert len(ctx.emitted) == 0 + + +def test_merged_decision_to_entry_body_shape() -> None: + """Verify the inline/sections shape for a merged decision.""" + result = _FakeAlgoResult( + owner_id=None, + title="merged title", + decision="merged text", + reason="merged reason", + impact="merged impact", + tags=["runtime"], + timestamp=1717200000000, + ) + inline, sections = _merged_decision_to_entry_body( + result, "cl_abc", "u_alice", "2026-06-01T00:00:00+00:00" + ) + assert inline["parent_type"] == "cluster" + assert inline["parent_id"] == "cl_abc" + assert inline["owner_id"] == "u_alice" + assert inline["tags"] == ["runtime"] + assert "session_id" not in inline + assert sections["Title"] == "merged title" + assert sections["Decision"] == "merged text" + assert sections["Reason"] == "merged reason" + assert sections["Impact"] == "merged impact" + + +def test_merged_decision_omits_empty_impact() -> None: + result = _FakeAlgoResult( + owner_id=None, + title="t", + decision="d", + reason="r", + timestamp=1717200000000, + ) + _, sections = _merged_decision_to_entry_body( + result, "cl_abc", "u_alice", "2026-06-01T00:00:00+00:00" + ) + assert "Impact" not in sections + + +def test_ts_to_ms_datetime() -> None: + dt = _dt.datetime(2026, 6, 1, tzinfo=_dt.UTC) + ms = _ts_to_ms(dt) + assert isinstance(ms, int) + assert ms > 0 + + +def test_ts_to_ms_int_passthrough() -> None: + assert _ts_to_ms(1717200000000) == 1717200000000 diff --git a/tests/unit/test_memory/test_search/test_adapter.py b/tests/unit/test_memory/test_search/test_adapter.py index 455089522..c979730be 100644 --- a/tests/unit/test_memory/test_search/test_adapter.py +++ b/tests/unit/test_memory/test_search/test_adapter.py @@ -42,6 +42,24 @@ def test_hybrid_skill_picks_skill_hybrid() -> None: assert fm == "skill_hybrid" +def test_hybrid_decision_picks_rrf() -> None: + fm, cfg = resolve_pipeline(SearchMethod.HYBRID, "decision") + assert fm == "rrf" + assert cfg is None + + +def test_keyword_decision_skips_everalgo() -> None: + fm, cfg = resolve_pipeline(SearchMethod.KEYWORD, "decision") + assert fm is None + assert cfg is None + + +def test_vector_decision_skips_everalgo() -> None: + fm, cfg = resolve_pipeline(SearchMethod.VECTOR, "decision") + assert fm is None + assert cfg is None + + def test_agentic_method_raises_value_error() -> None: """AGENTIC (a valid enum member) raises ValueError from resolve_pipeline. diff --git a/tests/unit/test_memory/test_search/test_dto.py b/tests/unit/test_memory/test_search/test_dto.py index f1a5a5676..727515a8f 100644 --- a/tests/unit/test_memory/test_search/test_dto.py +++ b/tests/unit/test_memory/test_search/test_dto.py @@ -40,6 +40,7 @@ def test_minimal_request_uses_hybrid_default() -> None: assert req.method == SearchMethod.HYBRID assert req.top_k == -1 assert req.include_profile is False + assert req.include_principles is False assert req.filters is None assert req.radius is None assert req.min_score is None @@ -137,11 +138,58 @@ def test_response_default_arrays_present() -> None: """Every ``data.*`` array must exist so callers can iterate unconditionally.""" resp = SearchResponse(request_id="0" * 32, data=SearchData()) assert resp.data.episodes == [] + assert resp.data.decisions == [] assert resp.data.profiles == [] + assert resp.data.principles == [] assert resp.data.agent_cases == [] assert resp.data.agent_skills == [] +def test_include_principles_defaults_to_false() -> None: + req = SearchRequest(**_minimal_request_kwargs()) + assert req.include_principles is False + + +def test_include_principles_independent_of_include_profile() -> None: + req = SearchRequest(**_minimal_request_kwargs(), include_principles=True) + assert req.include_principles is True + assert req.include_profile is False + req2 = SearchRequest(**_minimal_request_kwargs(), include_profile=True) + assert req2.include_principles is False + + +def test_kinds_defaults_to_none() -> None: + req = SearchRequest(**_minimal_request_kwargs()) + assert req.kinds is None + + +def test_kinds_accepts_episode_and_decision() -> None: + req = SearchRequest(**_minimal_request_kwargs(), kinds=["episode", "decision"]) + assert req.kinds == ["episode", "decision"] + assert SearchRequest(**_minimal_request_kwargs(), kinds=["decision"]).kinds == [ + "decision" + ] + assert SearchRequest(**_minimal_request_kwargs(), kinds=["episode"]).kinds == [ + "episode" + ] + + +def test_kinds_empty_list_rejected() -> None: + with pytest.raises(ValidationError, match="non-empty"): + SearchRequest(**_minimal_request_kwargs(), kinds=[]) + + +def test_kinds_principle_rejected() -> None: + """Principle is Meta Memory — not a searchable kind.""" + with pytest.raises(ValidationError): + SearchRequest(**_minimal_request_kwargs(), kinds=["principle"]) # type: ignore[list-item] + + +def test_kinds_rejected_when_agent_id_set() -> None: + with pytest.raises(ValidationError, match="kinds is only valid"): + SearchRequest(agent_id="agent_x", query="hello", kinds=["decision"]) + + def test_method_enum_serialises_to_lowercase() -> None: req = SearchRequest(**_minimal_request_kwargs(), method="agentic") # type: ignore[arg-type] assert req.method == SearchMethod.AGENTIC diff --git a/tests/unit/test_memory/test_search/test_filters.py b/tests/unit/test_memory/test_search/test_filters.py index 808c2a382..570e2925f 100644 --- a/tests/unit/test_memory/test_search/test_filters.py +++ b/tests/unit/test_memory/test_search/test_filters.py @@ -8,6 +8,7 @@ FilterError, FilterNode, compile_filters, + compile_filters_for_decision, ) # ── Base injection ─────────────────────────────────────────────────────── @@ -266,3 +267,13 @@ def test_compile_filters_excludes_deprecated_by_for_user() -> None: def test_compile_filters_omits_deprecated_by_for_agent() -> None: result = compile_filters(None, owner_id="agent_1", owner_type="agent") assert "deprecated_by" not in result + + +def test_compile_filters_for_decision_strips_sender_id_keeps_session() -> None: + node = FilterNode.model_validate({"sender_id": "u_jason", "session_id": "sess_a"}) + where = compile_filters_for_decision(node, owner_id="alice", owner_type="user") + assert "sender_id" not in where + assert "sender_ids" not in where + assert "session_id = 'sess_a'" in where + assert "deprecated_by IS NULL" in where + assert "owner_id = 'alice'" in where diff --git a/tests/unit/test_memory/test_search/test_manager.py b/tests/unit/test_memory/test_search/test_manager.py index 73dd66611..bb91b5cb7 100644 --- a/tests/unit/test_memory/test_search/test_manager.py +++ b/tests/unit/test_memory/test_search/test_manager.py @@ -28,7 +28,7 @@ from everos.component.embedding import EmbeddingCapability from everos.component.rerank import RerankCapability from everos.core.errors import ProviderNotConfiguredError -from everos.memory.search.dto import SearchMethod, SearchRequest +from everos.memory.search.dto import SearchMethod, SearchPrincipleItem, SearchRequest from everos.memory.search.manager import SearchManager # ── Stubs ─────────────────────────────────────────────────────────────── @@ -60,6 +60,25 @@ def _episode_row( ) +def _decision_row(did: str, score: float = 0.75) -> Candidate: + return Candidate( + id=did, + score=score, + source="keyword", + metadata={ + "owner_id": "alice", + "owner_type": "user", + "session_id": "sess_a", + "timestamp": _ts(), + "title": f"title {did}", + "decision": f"decision {did}", + "reason": f"reason {did}", + "impact": None, + "tags": ["runtime"], + }, + ) + + def _case_row(cid: str) -> Candidate: return Candidate( id=cid, @@ -103,11 +122,13 @@ class _StubEpisodeRecaller: def __init__(self, sparse: list[Candidate], dense: list[Candidate]) -> None: self._sparse = sparse self._dense = dense + self.sparse_calls = 0 self.last_where: str | None = None async def sparse_recall( self, query: str, where: str, *, limit: int ) -> list[Candidate]: + self.sparse_calls += 1 self.last_where = where return list(self._sparse[:limit]) @@ -163,6 +184,33 @@ async def facts_for_episodes( } +class _StubDecisionRecaller: + kind: ClassVar[str] = "decision" + everalgo_memory_type: ClassVar[str] = "" + text_field: ClassVar[str] = "decision" + + def __init__(self, sparse: list[Candidate], dense: list[Candidate]) -> None: + self._sparse = sparse + self._dense = dense + self.sparse_calls = 0 + self.dense_calls = 0 + self.last_where: str | None = None + + async def sparse_recall( + self, query: str, where: str, *, limit: int + ) -> list[Candidate]: + self.sparse_calls += 1 + self.last_where = where + return list(self._sparse[:limit]) + + async def dense_recall( + self, vector: Sequence[float], where: str, *, limit: int + ) -> list[Candidate]: + self.dense_calls += 1 + self.last_where = where + return list(self._dense[:limit]) + + class _StubAgentCaseRecaller: kind: ClassVar[str] = "agent_case" everalgo_memory_type: ClassVar[str] = "case" @@ -213,6 +261,22 @@ async def fetch(self, owner_id: str) -> list: return [] +class _StubPrincipleRecaller: + def __init__(self, items: list | None = None) -> None: + self._items = items or [] + self.fetch_calls: list[tuple[str, str, str]] = [] + + async def fetch( + self, + owner_id: str, + *, + app_id: str = "default", + project_id: str = "default", + ) -> list: + self.fetch_calls.append((owner_id, app_id, project_id)) + return list(self._items) + + class _StubEmbedding: def __init__(self, dim: int = 4) -> None: self.dim = dim @@ -231,6 +295,8 @@ def _build_manager( *, episode_sparse: list[Candidate] | None = None, episode_dense: list[Candidate] | None = None, + decision_sparse: list[Candidate] | None = None, + decision_dense: list[Candidate] | None = None, case_sparse: list[Candidate] | None = None, case_dense: list[Candidate] | None = None, skill_sparse: list[Candidate] | None = None, @@ -241,16 +307,21 @@ def _build_manager( embedding: _StubEmbedding | None = None, reranker: Any = None, llm_client: Any = None, + principle_recaller: _StubPrincipleRecaller | None = None, ) -> SearchManager: ep_recaller = _StubEpisodeRecaller(episode_sparse or [], episode_dense or []) return SearchManager( episode_recaller=ep_recaller, atomic_fact_recaller=_StubAtomicFactRecaller(facts_map, atomic_fact_dense), + decision_recaller=_StubDecisionRecaller( + decision_sparse or [], decision_dense or [] + ), agent_case_recaller=_StubAgentCaseRecaller(case_sparse or [], case_dense or []), agent_skill_recaller=_StubAgentSkillRecaller( skill_sparse or [], skill_dense or [], skill_by_case ), profile_recaller=_StubProfileRecaller(), + principle_recaller=principle_recaller or _StubPrincipleRecaller(), embedding=embedding, reranker=reranker, llm_client=llm_client, @@ -300,6 +371,7 @@ async def test_user_keyword_returns_episodes_only() -> None: assert resp.data.episodes[0].user_id == "alice" assert resp.data.episodes[0].type == "Conversation" # Agent paths stay empty. + assert resp.data.decisions == [] assert resp.data.agent_cases == [] assert resp.data.agent_skills == [] assert resp.data.profiles == [] @@ -384,9 +456,11 @@ async def test_user_keyword_filters_compile_pinned_owner() -> None: mgr = SearchManager( episode_recaller=recaller, atomic_fact_recaller=_StubAtomicFactRecaller(), + decision_recaller=_StubDecisionRecaller([], []), agent_case_recaller=_StubAgentCaseRecaller([], []), agent_skill_recaller=_StubAgentSkillRecaller([], []), profile_recaller=_StubProfileRecaller(), + principle_recaller=_StubPrincipleRecaller(), embedding=None, reranker=None, llm_client=None, @@ -397,6 +471,143 @@ async def test_user_keyword_filters_compile_pinned_owner() -> None: assert "owner_type = 'user'" in recaller.last_where +# ── Decision lane ────────────────────────────────────────────────────── + + +async def test_user_keyword_recalls_episodes_and_decisions() -> None: + """Default user KEYWORD runs episode + decision sparse lanes in parallel.""" + mgr = _build_manager( + episode_sparse=[_episode_row("ep_1")], + decision_sparse=[_decision_row("dc_1")], + ) + resp = await mgr.search(_user_req()) + assert [e.id for e in resp.data.episodes] == ["ep_1"] + assert [d.id for d in resp.data.decisions] == ["dc_1"] + assert mgr._ep.sparse_calls == 1 + assert mgr._decision.sparse_calls == 1 + + +async def test_kinds_decision_skips_episode_lane() -> None: + mgr = _build_manager( + episode_sparse=[_episode_row("ep_1")], + decision_sparse=[_decision_row("dc_1")], + ) + resp = await mgr.search(_user_req(kinds=["decision"])) + assert resp.data.episodes == [] + assert [d.id for d in resp.data.decisions] == ["dc_1"] + assert mgr._ep.sparse_calls == 0 + assert mgr._decision.sparse_calls == 1 + + +async def test_kinds_episode_skips_decision_lane() -> None: + mgr = _build_manager( + episode_sparse=[_episode_row("ep_1")], + decision_sparse=[_decision_row("dc_1")], + ) + resp = await mgr.search(_user_req(kinds=["episode"])) + assert [e.id for e in resp.data.episodes] == ["ep_1"] + assert resp.data.decisions == [] + assert mgr._ep.sparse_calls == 1 + assert mgr._decision.sparse_calls == 0 + + +async def test_user_keyword_decision_where_drops_sender_id() -> None: + """Mixed filters: episode keeps sender_id; decision lane strips it. + + Avoid a top-level ``session_id`` so ``_load_unprocessed`` stays off + (that path needs a real sqlite table this suite does not seed). + """ + from everos.memory.search.dto import FilterNode + + mgr = _build_manager( + episode_sparse=[_episode_row("ep_1")], + decision_sparse=[_decision_row("dc_1")], + ) + await mgr.search( + _user_req(filters=FilterNode.model_validate({"sender_id": "alice"})) + ) + assert mgr._ep.last_where is not None + assert "array_has(sender_ids, 'alice')" in mgr._ep.last_where + assert mgr._decision.last_where is not None + assert "sender_ids" not in mgr._decision.last_where + assert "deprecated_by IS NULL" in mgr._decision.last_where + + +async def test_user_hybrid_decision_does_not_call_arank( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Decision HYBRID is sparse+dense+rrf — never ``arank``.""" + calls: list[object] = [] + + async def _fake_arank(*args: Any, **kwargs: Any) -> Any: + calls.append((args, kwargs)) + raise AssertionError("decision HYBRID must not call arank") + + monkeypatch.setattr("everos.memory.search.manager.arank", _fake_arank) + mgr = _build_manager( + decision_sparse=[_decision_row("dc_1", score=0.8)], + decision_dense=[_decision_row("dc_1", score=0.7)], + embedding=_StubEmbedding(), + ) + resp = await mgr.search( + _user_req(method=SearchMethod.HYBRID, kinds=["decision"], top_k=5) + ) + assert calls == [] + assert [d.id for d in resp.data.decisions] == ["dc_1"] + assert resp.data.episodes == [] + + +async def test_user_agentic_still_fills_decisions_via_rrf( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """AGENTIC still only graphs episodes; decision lane remaps to rrf.""" + from everos.memory.search.dto import SearchEpisodeItem + + fake_result = [ + SearchEpisodeItem( + id="ep_1", + score=0.9, + session_id="s", + user_id="alice", + timestamp=_ts(), + sender_ids=["alice"], + subject="s", + summary="s", + episode="body", + type="Conversation", + atomic_facts=[], + ) + ] + + async def _fake_agentic(*args: Any, **kwargs: Any) -> list: + return fake_result + + monkeypatch.setattr( + "everos.memory.search.manager.search_episodes_agentic", _fake_agentic + ) + mgr = _build_manager( + decision_sparse=[_decision_row("dc_1")], + decision_dense=[_decision_row("dc_1")], + embedding=_StubEmbedding(), + reranker=_StubReranker(), + llm_client=_StubLLM(), + ) + resp = await mgr.search(_user_req(method=SearchMethod.AGENTIC)) + assert resp.data.episodes == fake_result + assert [d.id for d in resp.data.decisions] == ["dc_1"] + + +async def test_top_score_can_be_lifted_by_a_decision_hit() -> None: + mgr = _build_manager( + episode_sparse=[_episode_row("ep_1", score=0.2)], + decision_sparse=[_decision_row("dc_1", score=0.9)], + ) + resp = await mgr.search(_user_req()) + from everos.memory.search.manager import _top_score + + assert _top_score(resp.data) == pytest.approx(0.9) + + def _atomic_fact_row(fid: str, *, parent_id: str, score: float) -> Candidate: """Atomic-fact candidate emitted by ``AtomicFactRecaller.dense_recall``.""" return Candidate( @@ -758,6 +969,7 @@ async def test_agent_keyword_returns_cases_and_skills_only() -> None: ) resp = await mgr.search(_agent_req()) assert resp.data.episodes == [] + assert resp.data.decisions == [] assert resp.data.profiles == [] assert [c.id for c in resp.data.agent_cases] == ["c_1"] assert [s.id for s in resp.data.agent_skills] == ["s_1"] @@ -770,6 +982,76 @@ async def test_agent_owner_ignores_include_profile() -> None: assert resp.data.profiles == [] +async def test_include_principles_false_leaves_principles_empty() -> None: + item = SearchPrincipleItem( + id="u_alice_pr_aaaaaaaaaaaa", + user_id="alice", + title="Prefer Rust", + statement="Device Runtime is implemented in Rust.", + source_entry_ids=["dc_20260101_0001"], + timestamp=_ts(), + score=None, + ) + recaller = _StubPrincipleRecaller([item]) + mgr = _build_manager(principle_recaller=recaller) + resp = await mgr.search(_user_req()) + assert resp.data.principles == [] + assert recaller.fetch_calls == [] + + +async def test_include_principles_true_returns_kv_items() -> None: + item = SearchPrincipleItem( + id="u_alice_pr_aaaaaaaaaaaa", + user_id="alice", + title="Prefer Rust", + statement="Device Runtime is implemented in Rust.", + source_entry_ids=["dc_20260101_0001"], + timestamp=_ts(), + score=None, + ) + recaller = _StubPrincipleRecaller([item]) + mgr = _build_manager(principle_recaller=recaller) + resp = await mgr.search(_user_req(include_principles=True)) + assert [p.id for p in resp.data.principles] == ["u_alice_pr_aaaaaaaaaaaa"] + assert resp.data.principles[0].score is None + assert recaller.fetch_calls == [("alice", "default", "default")] + + +async def test_include_principles_independent_of_kinds() -> None: + item = SearchPrincipleItem( + id="u_alice_pr_aaaaaaaaaaaa", + user_id="alice", + title="Prefer Rust", + statement="Device Runtime is implemented in Rust.", + timestamp=_ts(), + ) + recaller = _StubPrincipleRecaller([item]) + mgr = _build_manager( + episode_sparse=[_episode_row("ep_1")], + decision_sparse=[_decision_row("dc_1")], + principle_recaller=recaller, + ) + resp = await mgr.search(_user_req(kinds=["episode"], include_principles=True)) + assert [e.id for e in resp.data.episodes] == ["ep_1"] + assert resp.data.decisions == [] + assert [p.title for p in resp.data.principles] == ["Prefer Rust"] + + +async def test_agent_owner_ignores_include_principles() -> None: + item = SearchPrincipleItem( + id="u_alice_pr_aaaaaaaaaaaa", + user_id="alice", + title="Prefer Rust", + statement="Device Runtime is implemented in Rust.", + timestamp=_ts(), + ) + recaller = _StubPrincipleRecaller([item]) + mgr = _build_manager(principle_recaller=recaller) + resp = await mgr.search(_agent_req(include_principles=True)) + assert resp.data.principles == [] + assert recaller.fetch_calls == [] + + # ── Top-k behaviour ─────────────────────────────────────────────────── @@ -1281,6 +1563,7 @@ async def test_search_captures_returned_hits_when_content_on( attrs = _span_index(_search_spans)["everos.memory.search"].attributes out = json.loads(attrs["langfuse.observation.output"]) assert out["episodes"] == ["ep_1"] + assert out["decisions"] == [] assert out["agent_cases"] == [] and out["agent_skills"] == [] diff --git a/tests/unit/test_memory/test_search/test_recall_principle.py b/tests/unit/test_memory/test_search/test_recall_principle.py new file mode 100644 index 000000000..0f3221ec8 --- /dev/null +++ b/tests/unit/test_memory/test_search/test_recall_principle.py @@ -0,0 +1,140 @@ +"""Real-LanceDB tests for ``PrincipleRecaller`` — KV-by-owner fetch. + +Principle recall has no query / no ranking: ``fetch(owner_id, app_id, +project_id)`` returns every row for that scope. These tests exercise +the LanceDB path (no stubs) and the owner / app / project isolation. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from everos.infra.persistence.lancedb import ( + Principle, + lancedb_manager, + principle_repo, +) +from everos.memory.search.recall.principle import PrincipleRecaller + + +def _principle_row( + *, + owner_id: str, + principle_id: str, + title: str = "Prefer Rust", + statement: str = "Device Runtime is implemented in Rust.", + source_entry_ids: list[str] | None = None, + timestamp_ms: int = 1_700_000_000_000, + app_id: str = "default", + project_id: str = "default", +) -> Principle: + return Principle( + id=f"{owner_id}_{principle_id}", + principle_id=principle_id, + owner_id=owner_id, + owner_type="user", + app_id=app_id, + project_id=project_id, + title=title, + statement=statement, + source_entry_ids=source_entry_ids or ["dc_20260101_0001"], + timestamp_ms=timestamp_ms, + md_path=f"users/{owner_id}/principles.md", + content_sha256="x" * 64, + ) + + +@pytest.fixture(autouse=True) +async def _reset(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("EVEROS_ROOT", str(tmp_path)) + lancedb_manager._conn = None + lancedb_manager._tables.clear() + yield + await lancedb_manager.dispose_connection() + + +async def test_fetch_returns_all_rows_for_owner() -> None: + await principle_repo.upsert( + [ + _principle_row( + owner_id="u_alice", + principle_id="pr_aaaaaaaaaaaa", + title="Prefer Rust", + ), + _principle_row( + owner_id="u_alice", + principle_id="pr_bbbbbbbbbbbb", + title="Keep Go control plane", + statement="Leave the control plane in Go.", + source_entry_ids=["dc_20260101_0002"], + ), + ] + ) + + items = await PrincipleRecaller().fetch("u_alice") + titles = {item.title for item in items} + assert titles == {"Prefer Rust", "Keep Go control plane"} + rust = next(i for i in items if i.title == "Prefer Rust") + assert rust.id == "u_alice_pr_aaaaaaaaaaaa" + assert rust.user_id == "u_alice" + assert rust.score is None + assert rust.source_entry_ids == ["dc_20260101_0001"] + assert rust.statement == "Device Runtime is implemented in Rust." + + +async def test_fetch_returns_empty_when_row_missing() -> None: + items = await PrincipleRecaller().fetch("u_cold_start") + assert items == [] + + +async def test_fetch_returns_empty_for_blank_owner() -> None: + items = await PrincipleRecaller().fetch("") + assert items == [] + + +async def test_fetch_isolates_by_owner() -> None: + await principle_repo.upsert( + [ + _principle_row( + owner_id="u_alice", + principle_id="pr_aaaaaaaaaaaa", + title="Alice", + ), + _principle_row( + owner_id="u_bob", + principle_id="pr_bbbbbbbbbbbb", + title="Bob", + ), + ] + ) + bob_items = await PrincipleRecaller().fetch("u_bob") + assert len(bob_items) == 1 + assert bob_items[0].title == "Bob" + + +async def test_fetch_isolates_by_app_project() -> None: + await principle_repo.upsert( + [ + _principle_row( + owner_id="u_alice", + principle_id="pr_aaaaaaaaaaaa", + title="Default space", + ), + _principle_row( + owner_id="u_alice", + principle_id="pr_cccccccccccc", + title="Other app", + app_id="other", + ), + ] + ) + items = await PrincipleRecaller().fetch( + "u_alice", app_id="default", project_id="default" + ) + assert [i.title for i in items] == ["Default space"] + other = await PrincipleRecaller().fetch( + "u_alice", app_id="other", project_id="default" + ) + assert [i.title for i in other] == ["Other app"] diff --git a/tests/unit/test_memory/test_search/test_shaper.py b/tests/unit/test_memory/test_search/test_shaper.py index 77c4bd3b4..a20afbf99 100644 --- a/tests/unit/test_memory/test_search/test_shaper.py +++ b/tests/unit/test_memory/test_search/test_shaper.py @@ -14,6 +14,7 @@ shape_agent_case_from_candidate, shape_agent_skill_from_candidate, shape_atomic_fact_from_candidate, + shape_decision_from_candidate, shape_episode_from_candidate, ) @@ -42,6 +43,25 @@ def _episode_candidate(*, id: str = "alice_ep_1", score: float = 0.9) -> Candida ) +def _decision_candidate(*, id: str = "alice_dc_1", score: float = 0.88) -> Candidate: + return Candidate( + id=id, + score=score, + source="keyword", + metadata={ + "owner_id": "alice", + "owner_type": "user", + "session_id": "sess_a", + "timestamp": _ts(), + "title": "Runtime language", + "decision": "Use Rust for the device Runtime", + "reason": "Need deterministic latency", + "impact": "Rewrite the hot path", + "tags": ["runtime", "rust"], + }, + ) + + def _agent_case_candidate() -> Candidate: return Candidate( id="agent_a_case_1", @@ -121,6 +141,37 @@ def test_shape_episode_attaches_facts() -> None: assert item.atomic_facts[0].content == "Alice prefers oat milk" +# ── Decision shaping ──────────────────────────────────────────────────── + + +def test_shape_decision_basic() -> None: + item = shape_decision_from_candidate(_decision_candidate()) + assert item is not None + assert item.id == "alice_dc_1" + assert item.user_id == "alice" + assert item.title == "Runtime language" + assert item.decision == "Use Rust for the device Runtime" + assert item.reason == "Need deterministic latency" + assert item.impact == "Rewrite the hot path" + assert item.tags == ["runtime", "rust"] + assert item.score == 0.88 + assert "sender_ids" not in item.model_dump() + + +def test_shape_decision_missing_impact_is_none() -> None: + cand = _decision_candidate() + del cand.metadata["impact"] + item = shape_decision_from_candidate(cand) + assert item is not None + assert item.impact is None + + +def test_shape_decision_drops_when_owner_type_wrong() -> None: + cand = _decision_candidate() + cand.metadata["owner_type"] = "agent" + assert shape_decision_from_candidate(cand) is None + + # ── Agent case / skill shaping ────────────────────────────────────────── diff --git a/tests/unit/test_memory/test_search/test_validate_components.py b/tests/unit/test_memory/test_search/test_validate_components.py index d8d380208..95dacec73 100644 --- a/tests/unit/test_memory/test_search/test_validate_components.py +++ b/tests/unit/test_memory/test_search/test_validate_components.py @@ -59,9 +59,11 @@ def _build_manager( return SearchManager( episode_recaller=None, # type: ignore[arg-type] atomic_fact_recaller=None, # type: ignore[arg-type] + decision_recaller=None, # type: ignore[arg-type] agent_case_recaller=None, # type: ignore[arg-type] agent_skill_recaller=None, # type: ignore[arg-type] profile_recaller=None, # type: ignore[arg-type] + principle_recaller=None, # type: ignore[arg-type] embedding=_StubEmbedding() if embedding_present else None, # type: ignore[arg-type] reranker=_StubReranker() if reranker_present else None, # type: ignore[arg-type] llm_client=_StubLLM() if llm_present else None, @@ -298,3 +300,17 @@ def test_keyword_never_requires_embed_or_rerank(manager: SearchManager) -> None: injected providers ``None``.""" req = SearchRequest(user_id="u1", query="q", method=SearchMethod.KEYWORD) manager._validate_components(req) # no raise + + +def test_agentic_kinds_decision_needs_embed_not_rerank_or_llm( + embed_available: None, +) -> None: + """``kinds=["decision"]`` remaps AGENTIC to rrf — embed only, no graph.""" + manager = _build_manager(embedding_present=True, llm_present=False) + req = SearchRequest( + user_id="u1", + query="q", + method=SearchMethod.AGENTIC, + kinds=["decision"], + ) + manager._validate_components(req) # must not raise diff --git a/tests/unit/test_memory/test_strategies/test_extract_decision.py b/tests/unit/test_memory/test_strategies/test_extract_decision.py new file mode 100644 index 000000000..931bd2a06 --- /dev/null +++ b/tests/unit/test_memory/test_strategies/test_extract_decision.py @@ -0,0 +1,365 @@ +from __future__ import annotations + +import datetime as _dt +import importlib +from unittest.mock import AsyncMock, patch + +import pytest +import structlog.testing +from everalgo.types import ( + ChatMessage, + Decision, + MemCell, + ToolCall, + ToolCallFunction, + ToolCallRequest, + ToolCallResult, +) + +from everos.core.persistence import EntryId +from everos.infra.ome.testing import FakeStrategyContext +from everos.memory.events import DecisionExtracted, UserPipelineStarted +from everos.memory.strategies.extract_decision import extract_decision + +mod = importlib.import_module("everos.memory.strategies.extract_decision") + + +def _two_user_memcell() -> MemCell: + return MemCell( + items=[ + ChatMessage( + id="m1", + role="user", + content="alice plans a trip", + timestamp=1_700_000_000_000, + sender_id="u_alice", + ), + ChatMessage( + id="m2", + role="user", + content="bob will buy tickets", + timestamp=1_700_000_001_000, + sender_id="u_bob", + ), + ChatMessage( + id="m3", + role="assistant", + content="sounds good", + timestamp=1_700_000_002_000, + sender_id="agent", + ), + ], + timestamp=1_700_000_002_000, + ) + + +def _algo_decision( + *, + title: str = "Use Rust on device", + decision: str = "Device Runtime uses Rust.", + reason: str = "Need deterministic latency.", + impact: str | None = None, + tags: list[str] | None = None, +) -> Decision: + return Decision( + owner_id=None, + title=title, + decision=decision, + reason=reason, + impact=impact, + tags=["runtime"] if tags is None else tags, + timestamp=1_700_000_000_000, + ) + + +def _event(memcell: MemCell | None = None) -> UserPipelineStarted: + return UserPipelineStarted( + memcell_id="mc_a", + session_id="s1", + memcell=memcell or _two_user_memcell(), + ) + + +def _eids(*seqs: int) -> list[EntryId]: + return [EntryId(prefix="dc", date=_dt.date(2026, 5, 17), seq=seq) for seq in seqs] + + +async def test_strategy_meta_is_attached() -> None: + meta = extract_decision.meta + assert meta.name == "extract_decision" + assert UserPipelineStarted in meta.trigger.on + assert meta.emits == frozenset({DecisionExtracted}) + assert meta.enabled is True + assert meta.max_retries == 2 + + +async def test_one_llm_call_fans_out_per_user_owner( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """One aextract (no sender_id), then md + emit per user owner.""" + monkeypatch.setattr(mod, "_writer", None, raising=False) + algo = [ + _algo_decision(title="A", decision="choose A", reason="r1"), + _algo_decision(title="B", decision="choose B", reason="r2"), + ] + event = _event() + with ( + patch( + "everos.memory.strategies.extract_decision.get_llm_client", + return_value=object(), + ), + patch( + "everos.memory.strategies.extract_decision.DecisionExtractor" + ) as mock_cls, + patch("everos.memory.strategies.extract_decision.DecisionWriter") as mock_wcls, + structlog.testing.capture_logs() as captured, + ): + mock_cls.return_value.aextract = AsyncMock(return_value=algo) + mock_wcls.return_value.append_entries = AsyncMock( + side_effect=[_eids(1, 2), _eids(1, 2)] + ) + ctx = FakeStrategyContext() + await extract_decision(event, ctx) + + assert mock_cls.return_value.aextract.await_count == 1 + call = mock_cls.return_value.aextract.await_args + assert call.args[0] is event.memcell + assert "sender_id" not in call.kwargs + assert call.kwargs.get("prompt") is None + + assert mock_wcls.return_value.append_entries.call_count == 2 + owners = [c.args[0] for c in mock_wcls.return_value.append_entries.call_args_list] + assert owners == ["u_alice", "u_bob"] + for c in mock_wcls.return_value.append_entries.call_args_list: + assert len(c.args[1]) == 2 + assert c.kwargs["app_id"] == "default" + assert c.kwargs["project_id"] == "default" + + emitted = [e for e in ctx.emitted if isinstance(e, DecisionExtracted)] + assert len(emitted) == 4 + assert [e.owner_id for e in emitted] == [ + "u_alice", + "u_alice", + "u_bob", + "u_bob", + ] + assert [e.decision_entry_id for e in emitted[:2]] == [ + _eids(1, 2)[0].format(), + _eids(1, 2)[1].format(), + ] + assert emitted[0].title == "A" + assert emitted[0].decision_text == "choose A" + assert emitted[0].reason == "r1" + assert emitted[0].source == "pipeline" + assert emitted[0].memcell_id == "mc_a" + assert emitted[0].session_id == "s1" + assert emitted[0].decision_timestamp_ms == 1_700_000_000_000 + + matching = [e for e in captured if e.get("event") == "decisions_extracted"] + assert matching, "expected decisions_extracted log line" + assert matching[0]["count"] == 4 + assert matching[0]["owner_ids"] == ["u_alice", "u_bob"] + + +async def test_writes_inline_and_sections( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(mod, "_writer", None, raising=False) + algo = [ + _algo_decision(impact=None, tags=["runtime"]), + _algo_decision( + title="Keep local-first", + decision="Markdown is SoT.", + reason="Cascade projects.", + impact="Keep Python in the agent runtime.", + tags=[], + ), + ] + event = UserPipelineStarted( + memcell_id="mc_a", + session_id="s1", + memcell=MemCell( + items=[ + ChatMessage( + id="m1", + role="user", + content="planning a trip", + timestamp=1_700_000_000_000, + sender_id="u_alice", + ) + ], + timestamp=1_700_000_000_000, + ), + ) + with ( + patch( + "everos.memory.strategies.extract_decision.get_llm_client", + return_value=object(), + ), + patch( + "everos.memory.strategies.extract_decision.DecisionExtractor" + ) as mock_cls, + patch("everos.memory.strategies.extract_decision.DecisionWriter") as mock_wcls, + ): + mock_cls.return_value.aextract = AsyncMock(return_value=algo) + mock_wcls.return_value.append_entries = AsyncMock(return_value=_eids(1, 2)) + await extract_decision(event, FakeStrategyContext()) + + assert mock_wcls.return_value.append_entries.call_count == 1 + batch = mock_wcls.return_value.append_entries.call_args + assert batch.args[0] == "u_alice" + items = batch.args[1] + assert len(items) == 2 + + inline0, sections0 = items[0] + assert inline0["owner_id"] == "u_alice" + assert inline0["session_id"] == "s1" + assert inline0["parent_type"] == "memcell" + assert inline0["parent_id"] == "mc_a" + assert inline0["tags"] == ["runtime"] + assert "sender_ids" not in inline0 + assert sections0 == { + "Title": "Use Rust on device", + "Decision": "Device Runtime uses Rust.", + "Reason": "Need deterministic latency.", + } + assert "Impact" not in sections0 + + inline1, sections1 = items[1] + assert inline1["tags"] == [] + assert sections1["Impact"] == "Keep Python in the agent runtime." + assert sections1["Title"] == "Keep local-first" + + +async def test_empty_algo_list_writes_and_emits_nothing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(mod, "_writer", None, raising=False) + event = _event() + with ( + patch( + "everos.memory.strategies.extract_decision.get_llm_client", + return_value=object(), + ), + patch( + "everos.memory.strategies.extract_decision.DecisionExtractor" + ) as mock_cls, + patch("everos.memory.strategies.extract_decision.DecisionWriter") as mock_wcls, + structlog.testing.capture_logs() as captured, + ): + mock_cls.return_value.aextract = AsyncMock(return_value=[]) + mock_wcls.return_value.append_entries = AsyncMock(return_value=[]) + ctx = FakeStrategyContext() + await extract_decision(event, ctx) + + assert mock_cls.return_value.aextract.await_count == 1 + mock_wcls.return_value.append_entries.assert_not_called() + assert ctx.emitted == [] + matching = [e for e in captured if e.get("event") == "decisions_extracted"] + assert matching[0]["count"] == 0 + + +async def test_skips_llm_when_memcell_has_no_user_senders( + monkeypatch: pytest.MonkeyPatch, +) -> None: + event = UserPipelineStarted( + memcell_id="mc_b", + session_id="s1", + memcell=MemCell(items=[], timestamp=1_700_000_000_000), + ) + monkeypatch.setattr(mod, "_writer", None, raising=False) + with ( + patch( + "everos.memory.strategies.extract_decision.get_llm_client", + return_value=object(), + ), + patch( + "everos.memory.strategies.extract_decision.DecisionExtractor" + ) as mock_cls, + patch("everos.memory.strategies.extract_decision.DecisionWriter") as mock_wcls, + structlog.testing.capture_logs() as captured, + ): + mock_cls.return_value.aextract = AsyncMock(return_value=[]) + mock_wcls.return_value.append_entries = AsyncMock(return_value=[]) + ctx = FakeStrategyContext() + await extract_decision(event, ctx) + + mock_cls.assert_not_called() + mock_wcls.return_value.append_entries.assert_not_called() + matching = [e for e in captured if e.get("event") == "decisions_extracted"] + assert matching, "log line should still fire (count=0)" + assert matching[0]["count"] == 0 + + +def _tool_call_memcell(*, with_user_message: bool) -> MemCell: + items: list[object] = [ + ToolCallRequest( + id="t1", + sender_id="agent", + timestamp=1_700_000_000_000, + tool_calls=[ + ToolCall( + id="c1", + function=ToolCallFunction(name="read_file", arguments="{}"), + ) + ], + ), + ToolCallResult( + id="t2", + timestamp=1_700_000_001_000, + tool_call_id="c1", + content="file contents", + ), + ] + if with_user_message: + items.insert( + 0, + ChatMessage( + id="m1", + role="user", + content="please fix the autoreloader", + timestamp=1_699_999_999_000, + sender_id="u_alice", + ), + ) + return MemCell(items=items, timestamp=1_700_000_000_000) + + +@pytest.mark.parametrize( + ("with_user_message", "expect_llm"), + [ + pytest.param(False, False, id="pure_agent_trajectory"), + pytest.param(True, True, id="mixed_user_and_tool_calls"), + ], +) +async def test_tool_calls_do_not_crash_the_sender_scan( + monkeypatch: pytest.MonkeyPatch, + with_user_message: bool, + expect_llm: bool, +) -> None: + event = UserPipelineStarted( + memcell_id="mc_tool", + session_id="s1", + memcell=_tool_call_memcell(with_user_message=with_user_message), + ) + monkeypatch.setattr(mod, "_writer", None, raising=False) + + with ( + patch( + "everos.memory.strategies.extract_decision.get_llm_client", + return_value=object(), + ), + patch( + "everos.memory.strategies.extract_decision.DecisionExtractor" + ) as mock_cls, + patch("everos.memory.strategies.extract_decision.DecisionWriter") as mock_wcls, + ): + mock_cls.return_value.aextract = AsyncMock(return_value=[]) + mock_wcls.return_value.append_entries = AsyncMock(return_value=[]) + await extract_decision(event, FakeStrategyContext()) + + if expect_llm: + assert mock_cls.return_value.aextract.await_count == 1 + assert "sender_id" not in mock_cls.return_value.aextract.await_args.kwargs + else: + mock_cls.assert_not_called() diff --git a/tests/unit/test_memory/test_strategies/test_extract_principles.py b/tests/unit/test_memory/test_strategies/test_extract_principles.py new file mode 100644 index 000000000..4c63a13bc --- /dev/null +++ b/tests/unit/test_memory/test_strategies/test_extract_principles.py @@ -0,0 +1,475 @@ +"""Tests for :func:`extract_principles`. + +Heavy mocking — the strategy threads through ``cluster_repo`` (sqlite +``kind=decision``), ``DecisionReader`` (md SoT), ``decision_repo`` +(Lance fallback), ``ProfileWriter`` (principles.md), and +``PrincipleExtractor`` (algo). We mock all seams so the test exercises +the orchestration only: union every cluster, snapshot the triggering +row, persist once. +""" + +from __future__ import annotations + +import asyncio +import importlib +from types import SimpleNamespace +from unittest.mock import AsyncMock, patch + +import numpy as np +import pytest +from everalgo.clustering import Cluster as AlgoCluster +from everalgo.types import Principle as AlgoPrinciple + +from everos.infra.ome.testing import FakeStrategyContext +from everos.infra.persistence.markdown import PrincipleFrontmatter +from everos.memory._partition_locks import _reset_for_tests +from everos.memory.events import DecisionClusterUpdated +from everos.memory.strategies.extract_principles import extract_principles +from everos.service.memorize import _STRATEGIES_ALWAYS, _STRATEGIES_REQUIRE_EMBED + +_MOD = "everos.memory.strategies.extract_principles" + + +@pytest.fixture(autouse=True) +def _isolate_partition_locks() -> None: + _reset_for_tests() + + +def _event( + *, + owner_id: str = "u_alice", + memcell_id: str = "mc_aaaaaaaaaaa1", + cluster_id: str = "cl_dec000000001", + decision_entry_id: str = "dc_20260101_0001", + title: str = "Use Rust runtime", + decision_text: str = "Device Runtime is implemented in Rust.", + reason: str = "Need deterministic latency.", + impact: str | None = None, + tags: list[str] | None = None, + decision_timestamp_ms: int = 1_700_000_001_000, +) -> DecisionClusterUpdated: + return DecisionClusterUpdated( + memcell_id=memcell_id, + decision_entry_id=decision_entry_id, + cluster_id=cluster_id, + owner_id=owner_id, + title=title, + decision_text=decision_text, + reason=reason, + impact=impact, + tags=list(tags or ["runtime"]), + decision_timestamp_ms=decision_timestamp_ms, + ) + + +def _algo_cluster(*, cluster_id: str, members: list[str]) -> AlgoCluster: + return AlgoCluster( + id=cluster_id, + centroid=np.zeros(1024, dtype=np.float32), + count=len(members), + last_ts=1_700_000_001_000, + preview=[], + members=members, + ) + + +def _principle( + *, title: str, statement: str, source_entry_ids: list[str] +) -> AlgoPrinciple: + return AlgoPrinciple( + owner_id="u_alice", + title=title, + statement=statement, + source_entry_ids=source_entry_ids, + timestamp=1_700_000_001_000, + ) + + +def _structured( + *, + title: str, + decision: str, + reason: str = "why", + tags: str = "[runtime]", + timestamp: str = "2026-01-01T00:00:00+00:00", +) -> SimpleNamespace: + return SimpleNamespace( + sections={"Title": title, "Decision": decision, "Reason": reason}, + inline={"tags": tags, "timestamp": timestamp}, + ) + + +async def test_strategy_meta_is_attached() -> None: + meta = extract_principles.meta + assert meta.name == "extract_principles" + assert DecisionClusterUpdated in meta.trigger.on + assert meta.emits == frozenset() + assert meta.max_retries == 2 + assert extract_principles in _STRATEGIES_ALWAYS + assert extract_principles not in _STRATEGIES_REQUIRE_EMBED + + +async def _run( + monkeypatch: pytest.MonkeyPatch, + event: DecisionClusterUpdated, + *, + clusters: list[AlgoCluster], + extractor: AsyncMock, + structured_by_id: dict[str, SimpleNamespace] | None = None, + lance_by_id: dict[str, object] | None = None, + mint_ids: list[str] | None = None, +) -> tuple[AsyncMock, AsyncMock]: + """Run extract_principles under the standard mock stack.""" + with ( + patch(f"{_MOD}.cluster_repo") as mock_cluster_repo, + patch(f"{_MOD}.DecisionReader") as mock_reader_cls, + patch(f"{_MOD}.decision_repo") as mock_decision_repo, + patch(f"{_MOD}.get_llm_client", return_value=object()), + patch(f"{_MOD}.PrincipleExtractor") as mock_extractor_cls, + patch(f"{_MOD}.ProfileWriter") as mock_writer_cls, + ): + mock_cluster_repo.list_for_owner = AsyncMock(return_value=clusters) + structured = structured_by_id or {} + + async def _find_structured(_owner: str, entry_id: str, **_kw: object): + return structured.get(entry_id) + + mock_reader_cls.return_value.find_structured = AsyncMock( + side_effect=_find_structured + ) + lance = lance_by_id or {} + + async def _find_lance(_owner: str, entry_id: str, **_kw: object): + return lance.get(entry_id) + + mock_decision_repo.find_by_owner_entry = AsyncMock(side_effect=_find_lance) + mock_extractor_cls.return_value.aextract = extractor + mock_writer_cls.return_value.write = AsyncMock(return_value=None) + + mod = importlib.import_module(_MOD) + monkeypatch.setattr(mod, "_writer", None, raising=False) + monkeypatch.setattr(mod, "_decision_reader", None, raising=False) + if mint_ids is not None: + ids = iter(mint_ids) + monkeypatch.setattr(mod, "mint_principle_id", lambda: next(ids)) + + await extract_principles(event, FakeStrategyContext()) + write = mock_writer_cls.return_value.write + aextract = mock_extractor_cls.return_value.aextract + return write, aextract + + +async def test_unions_all_decision_clusters_into_one_write( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Two clusters → extractor twice (once each); writer once with the union. + + Extracting only the triggering cluster and rewriting principles.md + would wipe cluster B — that is the would-wipe regression this pins. + """ + cluster_a = _algo_cluster(cluster_id="cl_a", members=["dc_20260101_0001"]) + cluster_b = _algo_cluster(cluster_id="cl_b", members=["dc_20260101_0002"]) + p_a = _principle( + title="From A", + statement="Prefer the snapshot cluster.", + source_entry_ids=["dc_20260101_0001"], + ) + p_b = _principle( + title="From B", + statement="Keep the other cluster.", + source_entry_ids=["dc_20260101_0002"], + ) + + async def fake_extract(decisions, *, owner_id, **_kw): + eid = decisions[0][0] + if eid == "dc_20260101_0001": + return [p_a] + return [p_b] + + write, aextract = await _run( + monkeypatch, + _event(), + clusters=[cluster_a, cluster_b], + extractor=AsyncMock(side_effect=fake_extract), + structured_by_id={ + "dc_20260101_0002": _structured(title="Other", decision="Keep cluster B.") + }, + mint_ids=["pr_aaaaaaaaaaaa", "pr_bbbbbbbbbbbb"], + ) + + assert aextract.await_count == 2 + write.assert_awaited_once() + kwargs = write.await_args.kwargs + fm = kwargs["frontmatter"] + assert isinstance(fm, PrincipleFrontmatter) + assert fm.id == "principle_u_alice" + assert fm.user_id == "u_alice" + titles = [item.title for item in fm.principles] + assert titles == ["From A", "From B"] + assert [item.id for item in fm.principles] == [ + "pr_aaaaaaaaaaaa", + "pr_bbbbbbbbbbbb", + ] + assert "Keep the other cluster." in kwargs["body"] + + +async def test_list_for_owner_uses_kind_decision( + monkeypatch: pytest.MonkeyPatch, +) -> None: + cluster = _algo_cluster(cluster_id="cl_a", members=["dc_20260101_0001"]) + with ( + patch(f"{_MOD}.cluster_repo") as mock_cluster_repo, + patch(f"{_MOD}.DecisionReader") as mock_reader_cls, + patch(f"{_MOD}.decision_repo") as mock_decision_repo, + patch(f"{_MOD}.get_llm_client", return_value=object()), + patch(f"{_MOD}.PrincipleExtractor") as mock_extractor_cls, + patch(f"{_MOD}.ProfileWriter") as mock_writer_cls, + ): + mock_cluster_repo.list_for_owner = AsyncMock(return_value=[cluster]) + mock_reader_cls.return_value.find_structured = AsyncMock(return_value=None) + mock_decision_repo.find_by_owner_entry = AsyncMock(return_value=None) + mock_extractor_cls.return_value.aextract = AsyncMock(return_value=[]) + mock_writer_cls.return_value.write = AsyncMock(return_value=None) + mod = importlib.import_module(_MOD) + monkeypatch.setattr(mod, "_writer", None, raising=False) + monkeypatch.setattr(mod, "_decision_reader", None, raising=False) + await extract_principles(_event(), FakeStrategyContext()) + + args, kwargs = mock_cluster_repo.list_for_owner.await_args + assert args[0] == "u_alice" + assert args[1] == "decision" + assert kwargs["app_id"] == "default" + assert kwargs["project_id"] == "default" + + +async def test_triggering_member_uses_event_snapshot_when_reader_misses( + monkeypatch: pytest.MonkeyPatch, +) -> None: + cluster = _algo_cluster(cluster_id="cl_a", members=["dc_20260101_0001"]) + captured: list[list] = [] + + async def fake_extract(decisions, *, owner_id, **_kw): + captured.append(list(decisions)) + return [] + + await _run( + monkeypatch, + _event(decision_text="Device Runtime is implemented in Rust."), + clusters=[cluster], + extractor=AsyncMock(side_effect=fake_extract), + structured_by_id={}, + lance_by_id={}, + ) + + assert len(captured) == 1 + eid, decision = captured[0][0] + assert eid == "dc_20260101_0001" + assert decision.decision == "Device Runtime is implemented in Rust." + assert decision.title == "Use Rust runtime" + assert decision.tags == ["runtime"] + + +async def test_other_members_load_via_decision_reader( + monkeypatch: pytest.MonkeyPatch, +) -> None: + cluster = _algo_cluster( + cluster_id="cl_a", + members=["dc_20260101_0001", "dc_20260101_0002"], + ) + captured: list[list] = [] + + async def fake_extract(decisions, *, owner_id, **_kw): + captured.append(list(decisions)) + return [] + + await _run( + monkeypatch, + _event(), + clusters=[cluster], + extractor=AsyncMock(side_effect=fake_extract), + structured_by_id={ + "dc_20260101_0002": _structured( + title="Keep Go", + decision="Leave the control plane in Go.", + ) + }, + ) + + by_id = {eid: dc for eid, dc in captured[0]} + assert by_id["dc_20260101_0001"].decision.startswith("Device Runtime") + assert by_id["dc_20260101_0002"].decision == "Leave the control plane in Go." + assert by_id["dc_20260101_0002"].title == "Keep Go" + + +async def test_empty_extractor_list_still_writes( + monkeypatch: pytest.MonkeyPatch, +) -> None: + cluster = _algo_cluster(cluster_id="cl_a", members=["dc_20260101_0001"]) + write, aextract = await _run( + monkeypatch, + _event(), + clusters=[cluster], + extractor=AsyncMock(return_value=[]), + ) + aextract.assert_awaited_once() + write.assert_awaited_once() + fm = write.await_args.kwargs["frontmatter"] + assert fm.principles == [] + + +async def test_no_clusters_writes_empty_file( + monkeypatch: pytest.MonkeyPatch, +) -> None: + write, aextract = await _run( + monkeypatch, + _event(), + clusters=[], + extractor=AsyncMock( + return_value=[ + _principle( + title="stale", + statement="should not run", + source_entry_ids=["x"], + ) + ] + ), + ) + aextract.assert_not_awaited() + write.assert_awaited_once() + assert write.await_args.kwargs["frontmatter"].principles == [] + + +async def test_unloadable_members_skip_write( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Members exist but none load (no snapshot text, no md, no lance).""" + cluster = _algo_cluster(cluster_id="cl_a", members=["dc_20260101_0001"]) + write, aextract = await _run( + monkeypatch, + _event(decision_text=""), + clusters=[cluster], + extractor=AsyncMock(return_value=[]), + ) + aextract.assert_not_awaited() + write.assert_not_awaited() + + +async def test_does_not_write_if_a_cluster_extract_raises( + monkeypatch: pytest.MonkeyPatch, +) -> None: + cluster_a = _algo_cluster(cluster_id="cl_a", members=["dc_20260101_0001"]) + cluster_b = _algo_cluster(cluster_id="cl_b", members=["dc_20260101_0002"]) + with ( + patch(f"{_MOD}.cluster_repo") as mock_cluster_repo, + patch(f"{_MOD}.DecisionReader") as mock_reader_cls, + patch(f"{_MOD}.decision_repo") as mock_decision_repo, + patch(f"{_MOD}.get_llm_client", return_value=object()), + patch(f"{_MOD}.PrincipleExtractor") as mock_extractor_cls, + patch(f"{_MOD}.ProfileWriter") as mock_writer_cls, + ): + mock_cluster_repo.list_for_owner = AsyncMock( + return_value=[cluster_a, cluster_b] + ) + mock_reader_cls.return_value.find_structured = AsyncMock( + return_value=_structured(title="B", decision="Keep B.") + ) + mock_decision_repo.find_by_owner_entry = AsyncMock(return_value=None) + mock_extractor_cls.return_value.aextract = AsyncMock( + side_effect=[ + [ + _principle( + title="A", + statement="a", + source_entry_ids=["dc_20260101_0001"], + ) + ], + RuntimeError("llm down"), + ] + ) + mock_writer_cls.return_value.write = AsyncMock(return_value=None) + mod = importlib.import_module(_MOD) + monkeypatch.setattr(mod, "_writer", None, raising=False) + monkeypatch.setattr(mod, "_decision_reader", None, raising=False) + with pytest.raises(RuntimeError, match="llm down"): + await extract_principles(_event(), FakeStrategyContext()) + mock_writer_cls.return_value.write.assert_not_awaited() + + +async def _run_serialisation_probe( + owner_a: str, owner_b: str, monkeypatch: pytest.MonkeyPatch +) -> list[str]: + log: list[str] = [] + + async def mock_aextract(decisions, *, owner_id, **_kwargs): + log.append(f"enter:{owner_id}") + await asyncio.sleep(0.01) + log.append(f"leave:{owner_id}") + return [] + + cluster_a = _algo_cluster(cluster_id="cl_a", members=["dc_20260101_0001"]) + cluster_b = _algo_cluster(cluster_id="cl_b", members=["dc_20260101_0002"]) + + with ( + patch(f"{_MOD}.cluster_repo") as mock_cluster_repo, + patch(f"{_MOD}.DecisionReader") as mock_reader_cls, + patch(f"{_MOD}.decision_repo") as mock_decision_repo, + patch(f"{_MOD}.get_llm_client", return_value=object()), + patch(f"{_MOD}.PrincipleExtractor") as mock_extractor_cls, + patch(f"{_MOD}.ProfileWriter") as mock_writer_cls, + ): + mock_cluster_repo.list_for_owner = AsyncMock( + side_effect=lambda owner, _kind, **_kw: ( + [cluster_a] if owner == owner_a else [cluster_b] + ) + ) + mock_reader_cls.return_value.find_structured = AsyncMock(return_value=None) + mock_decision_repo.find_by_owner_entry = AsyncMock(return_value=None) + mock_extractor_cls.return_value.aextract = mock_aextract + mock_writer_cls.return_value.write = AsyncMock(return_value=None) + + mod = importlib.import_module(_MOD) + monkeypatch.setattr(mod, "_writer", None, raising=False) + monkeypatch.setattr(mod, "_decision_reader", None, raising=False) + + await asyncio.gather( + extract_principles( + _event( + owner_id=owner_a, + cluster_id="cl_a", + decision_entry_id="dc_20260101_0001", + decision_text="A", + ), + FakeStrategyContext(), + ), + extract_principles( + _event( + owner_id=owner_b, + cluster_id="cl_a" if owner_b == owner_a else "cl_b", + decision_entry_id=( + "dc_20260101_0001" + if owner_b == owner_a + else "dc_20260101_0002" + ), + decision_text="B", + ), + FakeStrategyContext(), + ), + ) + return log + + +async def test_partition_lock_serialises_runs_on_same_owner( + monkeypatch: pytest.MonkeyPatch, +) -> None: + log = await _run_serialisation_probe("u_alice", "u_alice", monkeypatch) + assert log[0].startswith("enter:") and log[1].startswith("leave:") + assert log[2].startswith("enter:") and log[3].startswith("leave:") + + +async def test_partition_lock_lets_different_owners_run_in_parallel( + monkeypatch: pytest.MonkeyPatch, +) -> None: + log = await _run_serialisation_probe("u_alice", "u_bob", monkeypatch) + assert sorted(log) == sorted( + ["enter:u_alice", "leave:u_alice", "enter:u_bob", "leave:u_bob"] + ) diff --git a/tests/unit/test_memory/test_strategies/test_reflect_decisions.py b/tests/unit/test_memory/test_strategies/test_reflect_decisions.py new file mode 100644 index 000000000..b7d4f91cf --- /dev/null +++ b/tests/unit/test_memory/test_strategies/test_reflect_decisions.py @@ -0,0 +1,75 @@ +"""Tests for the ``reflect_decisions`` Cron strategy. + +Verifies decorator metadata (name, trigger type, emits, enabled flag). +The strategy body is a thin entry point — orchestrator logic is tested +separately in ``test_reflection/test_decision_orchestrator.py``. +""" + +from __future__ import annotations + +import inspect +from unittest.mock import AsyncMock, patch + +import structlog.testing + +from everos.component.embedding import EmbeddingCapability +from everos.infra.ome.testing import FakeStrategyContext +from everos.infra.ome.triggers import Cron +from everos.memory.events import DecisionExtracted +from everos.memory.strategies.reflect_decisions import reflect_decisions +from everos.service.memorize import _STRATEGIES_ALWAYS, _STRATEGIES_REQUIRE_EMBED + + +async def test_strategy_meta_is_attached() -> None: + """Decorator stamps the expected StrategyMeta on the function.""" + meta = reflect_decisions.meta + assert meta.name == "reflect_decisions" + assert isinstance(meta.trigger, Cron) + assert meta.trigger.expr == "0 2 * * 1" + assert meta.emits == frozenset({DecisionExtracted}) + assert meta.max_retries == 1 + assert meta.enabled is False + assert reflect_decisions in _STRATEGIES_REQUIRE_EMBED + assert reflect_decisions not in _STRATEGIES_ALWAYS + + +async def test_strategy_is_callable() -> None: + """The Strategy wrapper must be callable (delegates to async func).""" + assert callable(reflect_decisions) + assert inspect.iscoroutinefunction(reflect_decisions.meta.func) + + +async def test_returns_without_side_effects_when_embedding_unavailable() -> None: + """Capability unavailable → early return; no cluster iteration. + + Reflection re-embeds merged decision text, so it cannot run without + an embedder. Registration is unconditional so a runtime tier upgrade + (Tier 1 → Tier 2) picks up on the next scheduled tick; a Tier-1 + cron tick simply no-ops. + """ + from everos.infra.ome.events import CronTick + + with ( + patch( + "everos.memory.strategies.reflect_decisions.get_embedding_capability", + return_value=EmbeddingCapability(provider=None), + ), + patch( + "everos.memory.strategies.reflect_decisions.cluster_repo" + ) as mock_cluster_repo, + structlog.testing.capture_logs() as captured, + ): + mock_cluster_repo.list_distinct_owners = AsyncMock( + side_effect=AssertionError("cluster_repo must not be touched"), + ) + await reflect_decisions( + CronTick(strategy_name="reflect_decisions"), FakeStrategyContext() + ) + + gated = [ + e + for e in captured + if e.get("event") == "strategy_gated_off_embedding_unavailable" + ] + assert len(gated) == 1 + assert gated[0]["strategy_name"] == "reflect_decisions" diff --git a/tests/unit/test_memory/test_strategies/test_registration.py b/tests/unit/test_memory/test_strategies/test_registration.py index b070b81b4..9d8600711 100644 --- a/tests/unit/test_memory/test_strategies/test_registration.py +++ b/tests/unit/test_memory/test_strategies/test_registration.py @@ -11,9 +11,13 @@ extract_agent_case, extract_agent_skill, extract_atomic_facts, + extract_decision, extract_foresight, + extract_principles, extract_user_profile, + reflect_decisions, reflect_episodes, + trigger_decision_clustering, trigger_profile_clustering, trigger_skill_clustering, ) @@ -22,12 +26,16 @@ def test_strategies_are_re_exported_from_package() -> None: for fn, name in [ (extract_atomic_facts, "extract_atomic_facts"), + (extract_decision, "extract_decision"), (extract_foresight, "extract_foresight"), (extract_agent_case, "extract_agent_case"), (trigger_skill_clustering, "trigger_skill_clustering"), (extract_agent_skill, "extract_agent_skill"), (trigger_profile_clustering, "trigger_profile_clustering"), + (trigger_decision_clustering, "trigger_decision_clustering"), (extract_user_profile, "extract_user_profile"), + (extract_principles, "extract_principles"), + (reflect_decisions, "reflect_decisions"), (reflect_episodes, "reflect_episodes"), ]: assert fn.meta.name == name @@ -49,11 +57,15 @@ async def test_get_engine_registers_all_strategies( names = {m.name for m in engine._registry.all()} assert names == { "extract_atomic_facts", + "extract_decision", "extract_foresight", "extract_agent_case", "trigger_skill_clustering", "extract_agent_skill", "trigger_profile_clustering", + "trigger_decision_clustering", "extract_user_profile", + "extract_principles", + "reflect_decisions", "reflect_episodes", } diff --git a/tests/unit/test_memory/test_strategies/test_strategies_persistence.py b/tests/unit/test_memory/test_strategies/test_strategies_persistence.py index 678e574bc..4121f857b 100644 --- a/tests/unit/test_memory/test_strategies/test_strategies_persistence.py +++ b/tests/unit/test_memory/test_strategies/test_strategies_persistence.py @@ -7,13 +7,21 @@ from unittest.mock import AsyncMock, patch import pytest -from everalgo.types import AgentCase, AtomicFact, ChatMessage, Foresight, MemCell +from everalgo.types import ( + AgentCase, + AtomicFact, + ChatMessage, + Decision, + Foresight, + MemCell, +) from everos.core.persistence import MemoryRoot from everos.infra.ome.testing import FakeStrategyContext from everos.infra.persistence.markdown import ( AgentCaseReader, AtomicFactReader, + DecisionReader, ForesightReader, ) from everos.memory.events import ( @@ -23,6 +31,7 @@ ) from everos.memory.strategies.extract_agent_case import extract_agent_case from everos.memory.strategies.extract_atomic_facts import extract_atomic_facts +from everos.memory.strategies.extract_decision import extract_decision from everos.memory.strategies.extract_foresight import extract_foresight @@ -170,6 +179,52 @@ async def test_foresights_round_trip( assert "said so" in content +async def test_decisions_round_trip( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + import importlib + + dc_mod = importlib.import_module("everos.memory.strategies.extract_decision") + + monkeypatch.setattr( + MemoryRoot, "resolve", classmethod(lambda cls: MemoryRoot(root=tmp_path)) + ) + monkeypatch.setattr(dc_mod, "_writer", None, raising=False) + + decisions = [ + Decision( + owner_id=None, + title="Use Rust on device", + decision="Device Runtime uses Rust.", + reason="Need deterministic latency on the edge.", + impact=None, + tags=["runtime"], + timestamp=1_700_000_000_000, + ), + ] + + with ( + patch( + "everos.memory.strategies.extract_decision.get_llm_client", + return_value=object(), + ), + patch( + "everos.memory.strategies.extract_decision.DecisionExtractor" + ) as mock_ext, + ): + mock_ext.return_value.aextract = AsyncMock(return_value=decisions) + await extract_decision(_event_for("u_alice"), FakeStrategyContext()) + + reader = DecisionReader(root=MemoryRoot(root=tmp_path)) + path = reader.path_for("u_alice") + assert path.is_file(), f"expected md at {path}" + content = path.read_text(encoding="utf-8") + assert "Use Rust on device" in content + assert "Device Runtime uses Rust." in content + assert "Need deterministic latency on the edge." in content + + async def test_agent_case_round_trip( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, diff --git a/tests/unit/test_memory/test_strategies/test_strategy_to_handler_contract.py b/tests/unit/test_memory/test_strategies/test_strategy_to_handler_contract.py index db323d359..3bc48ed58 100644 --- a/tests/unit/test_memory/test_strategies/test_strategy_to_handler_contract.py +++ b/tests/unit/test_memory/test_strategies/test_strategy_to_handler_contract.py @@ -19,7 +19,14 @@ import anyio import pytest -from everalgo.types import AgentCase, AtomicFact, ChatMessage, Foresight, MemCell +from everalgo.types import ( + AgentCase, + AtomicFact, + ChatMessage, + Decision, + Foresight, + MemCell, +) from everos.component.embedding import EmbeddingCapability, EmbeddingProvider from everos.component.tokenizer import Tokenizer @@ -28,6 +35,7 @@ from everos.memory.cascade.handlers import ( AgentCaseHandler, AtomicFactHandler, + DecisionHandler, ForesightHandler, HandlerDeps, ) @@ -39,6 +47,7 @@ ) from everos.memory.strategies.extract_agent_case import extract_agent_case from everos.memory.strategies.extract_atomic_facts import extract_atomic_facts +from everos.memory.strategies.extract_decision import extract_decision from everos.memory.strategies.extract_foresight import extract_foresight @@ -103,7 +112,7 @@ def _event(owner_id: str) -> UserPipelineStarted: async def _build_row_from_md( - handler: AtomicFactHandler | ForesightHandler | AgentCaseHandler, + handler: AtomicFactHandler | ForesightHandler | AgentCaseHandler | DecisionHandler, md_root: Path, md_glob: str, *, @@ -224,6 +233,62 @@ async def test_foresight_strategy_md_feeds_handler_with_content( assert len(row.vector) == 1024 +async def test_decision_strategy_md_feeds_handler_with_content( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Strategy → md → DecisionHandler must carry title / decision / reason. + + Guards lowercase section-key drift (``sections.get("decision")``) + which would upsert a Lance row with empty ``decision`` text. + """ + dc_mod = importlib.import_module("everos.memory.strategies.extract_decision") + monkeypatch.setattr( + MemoryRoot, "resolve", classmethod(lambda cls: MemoryRoot(root=tmp_path)) + ) + monkeypatch.setattr(dc_mod, "_writer", None, raising=False) + + decisions = [ + Decision( + owner_id=None, + title="Use Rust for the device Runtime", + decision="Ship the device Runtime in Rust.", + reason="Need deterministic latency without a GC pause.", + impact=None, + tags=["runtime", "rust"], + timestamp=1_700_000_000_000, + ), + ] + with ( + patch( + "everos.memory.strategies.extract_decision.get_llm_client", + return_value=object(), + ), + patch( + "everos.memory.strategies.extract_decision.DecisionExtractor" + ) as mock_ext, + ): + mock_ext.return_value.aextract = AsyncMock(return_value=decisions) + await extract_decision(_event("u_alice"), FakeStrategyContext()) + + handler = DecisionHandler( + HandlerDeps( + memory_root=MemoryRoot(root=tmp_path), + tokenizer=_StubTokenizer(), + ) + ) + row = await _build_row_from_md( + handler, tmp_path, "*/*/users/u_alice/decisions/decision-*.md" + ) + assert row.title == "Use Rust for the device Runtime" + assert row.decision == "Ship the device Runtime in Rust." + assert row.decision_tokens == "Ship the device Runtime in Rust." + assert row.reason == "Need deterministic latency without a GC pause." + assert row.reason_tokens == "Need deterministic latency without a GC pause." + assert row.impact is None + assert row.tags == ["runtime", "rust"] + assert len(row.vector) == 1024 + + def _agent_event() -> AgentPipelineStarted: return AgentPipelineStarted( memcell_id="mc_a", diff --git a/tests/unit/test_memory/test_strategies/test_trigger_decision_clustering.py b/tests/unit/test_memory/test_strategies/test_trigger_decision_clustering.py new file mode 100644 index 000000000..aa067e7d5 --- /dev/null +++ b/tests/unit/test_memory/test_strategies/test_trigger_decision_clustering.py @@ -0,0 +1,316 @@ +"""Tests for :func:`trigger_decision_clustering`. + +Mirrors :mod:`test_trigger_profile_clustering`: mock embedder + +cluster_repo + cluster_by_geometry, drive the strategy via +:class:`FakeStrategyContext`, verify a single +:class:`DecisionClusterUpdated` event is emitted with the sqlite +``kind=decision`` / ``member_type=decision`` write path. +""" + +from __future__ import annotations + +import asyncio +from unittest.mock import AsyncMock, MagicMock, patch + +import numpy as np +import pytest +import structlog.testing +from everalgo.clustering import Cluster as AlgoCluster + +from everos.component.embedding import EmbeddingCapability, EmbeddingProvider +from everos.infra.ome.testing import FakeStrategyContext +from everos.memory._partition_locks import _reset_for_tests +from everos.memory.events import DecisionClusterUpdated, DecisionExtracted +from everos.memory.strategies.trigger_decision_clustering import ( + trigger_decision_clustering, +) + + +def _install_embedder( + monkeypatch: pytest.MonkeyPatch, embedder: EmbeddingProvider +) -> None: + """Install ``embedder`` as the process-wide embedding capability.""" + import everos.component.embedding.accessor as acc + + monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=embedder)) + + +@pytest.fixture(autouse=True) +def _isolate_partition_locks() -> None: + _reset_for_tests() + + +def _event( + *, + owner_id: str = "u_alice", + memcell_id: str = "mc_aaaaaaaaaaa1", + decision_entry_id: str = "dc_20260517_0001", + title: str = "Use Rust on device", + decision_text: str = "Device Runtime uses Rust.", + reason: str = "Need deterministic latency.", + impact: str | None = "Keep Python in the agent runtime.", + tags: list[str] | None = None, + decision_timestamp_ms: int = 1_700_000_001_000, + source: str = "pipeline", +) -> DecisionExtracted: + return DecisionExtracted( + memcell_id=memcell_id, + decision_entry_id=decision_entry_id, + title=title, + decision_text=decision_text, + reason=reason, + impact=impact, + tags=tags if tags is not None else ["runtime"], + decision_timestamp_ms=decision_timestamp_ms, + owner_id=owner_id, + session_id="s_test", + source=source, + ) + + +async def test_strategy_meta_is_attached() -> None: + meta = trigger_decision_clustering.meta + assert meta.name == "trigger_decision_clustering" + assert DecisionExtracted in meta.trigger.on + assert meta.emits == frozenset({DecisionClusterUpdated}) + assert meta.max_retries == 2 + assert meta.applies_to is not None + + +@pytest.mark.asyncio +async def test_creates_new_cluster_when_no_existing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Empty existing → cluster_by_geometry returns None → new cluster persisted.""" + embedder = MagicMock() + embedder.embed = AsyncMock(return_value=[0.1] * 1024) + _install_embedder(monkeypatch, embedder) + ctx = FakeStrategyContext() + + with ( + patch( + "everos.memory.strategies.trigger_decision_clustering.cluster_repo" + ) as mock_repo, + patch( + "everos.memory.strategies.trigger_decision_clustering.cluster_by_geometry", + new=MagicMock(return_value=None), + ) as mock_cluster, + patch( + "everos.memory.strategies.trigger_decision_clustering.mint_cluster_id", + return_value="cl_newdec000001", + ), + structlog.testing.capture_logs() as captured, + ): + mock_repo.list_for_owner = AsyncMock(return_value=[]) + mock_repo.upsert_with_members = AsyncMock(return_value=None) + + await trigger_decision_clustering(_event(), ctx) + + args, _ = mock_cluster.call_args + new_cluster, existing = args + assert isinstance(new_cluster, AlgoCluster) + assert new_cluster.id == "cl_newdec000001" + assert new_cluster.count == 1 + assert new_cluster.last_ts == 1_700_000_001_000 + assert new_cluster.members == ["dc_20260517_0001"] + assert new_cluster.preview == ["Device Runtime uses Rust."] + assert existing == [] + + mock_repo.list_for_owner.assert_awaited_once_with( + "u_alice", + "decision", + app_id="default", + project_id="default", + ) + + upsert_args = mock_repo.upsert_with_members.call_args + persisted = upsert_args.args[0] + assert persisted.id == "cl_newdec000001" + assert upsert_args.kwargs == { + "owner_id": "u_alice", + "owner_type": "user", + "kind": "decision", + "member_type": "decision", + "app_id": "default", + "project_id": "default", + } + + emitted = [e for e in ctx.emitted if isinstance(e, DecisionClusterUpdated)] + assert len(emitted) == 1 + assert emitted[0].memcell_id == "mc_aaaaaaaaaaa1" + assert emitted[0].decision_entry_id == "dc_20260517_0001" + assert emitted[0].cluster_id == "cl_newdec000001" + assert emitted[0].owner_id == "u_alice" + assert emitted[0].title == "Use Rust on device" + assert emitted[0].decision_text == "Device Runtime uses Rust." + assert emitted[0].reason == "Need deterministic latency." + assert emitted[0].impact == "Keep Python in the agent runtime." + assert emitted[0].tags == ["runtime"] + assert emitted[0].decision_timestamp_ms == 1_700_000_001_000 + + matching = [r for r in captured if r.get("event") == "decision_cluster_updated"] + assert matching, "expected decision_cluster_updated log line" + + +@pytest.mark.asyncio +async def test_merges_into_existing_cluster_when_algo_matches( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """algo returns merged Cluster → persisted under the existing id.""" + embedder = MagicMock() + embedder.embed = AsyncMock(return_value=[0.2] * 1024) + _install_embedder(monkeypatch, embedder) + ctx = FakeStrategyContext() + + existing_cluster = AlgoCluster( + id="cl_existing0001", + centroid=np.array([0.15] * 1024, dtype=np.float32), + count=1, + last_ts=1_700_000_000_000, + preview=["earlier decision"], + members=["dc_20260517_0000"], + ) + merged_cluster = AlgoCluster( + id="cl_existing0001", + centroid=np.array([0.17] * 1024, dtype=np.float32), + count=2, + last_ts=1_700_000_001_000, + preview=["earlier decision", "Device Runtime uses Rust."], + members=["dc_20260517_0000", "dc_20260517_0001"], + ) + + with ( + patch( + "everos.memory.strategies.trigger_decision_clustering.cluster_repo" + ) as mock_repo, + patch( + "everos.memory.strategies.trigger_decision_clustering.cluster_by_geometry", + new=MagicMock(return_value=merged_cluster), + ), + ): + mock_repo.list_for_owner = AsyncMock(return_value=[existing_cluster]) + mock_repo.upsert_with_members = AsyncMock(return_value=None) + + await trigger_decision_clustering(_event(), ctx) + + persisted = mock_repo.upsert_with_members.call_args.args[0] + assert persisted.id == "cl_existing0001" + assert persisted.count == 2 + + emitted = [e for e in ctx.emitted if isinstance(e, DecisionClusterUpdated)] + assert len(emitted) == 1 + assert emitted[0].cluster_id == "cl_existing0001" + + +async def _run_serialisation_probe( + owner_a: str, owner_b: str, monkeypatch: pytest.MonkeyPatch +) -> list[str]: + """Drive two trigger_decision_clustering runs and record entry/exit order.""" + log: list[str] = [] + + def mock_cluster_by_geometry(_new_cluster, _existing, **_kw): + return None + + async def mock_upsert(cluster, **_kwargs): + mid = cluster.members[0] + log.append(f"enter:{mid}") + await asyncio.sleep(0.01) + log.append(f"leave:{mid}") + + mock_embedder = MagicMock() + mock_embedder.embed = AsyncMock(return_value=np.zeros(1024, dtype=np.float32)) + _install_embedder(monkeypatch, mock_embedder) + + with ( + patch( + "everos.memory.strategies.trigger_decision_clustering.cluster_repo" + ) as mock_repo, + patch( + "everos.memory.strategies.trigger_decision_clustering.cluster_by_geometry", + new=mock_cluster_by_geometry, + ), + ): + mock_repo.list_for_owner = AsyncMock(return_value=[]) + mock_repo.upsert_with_members = mock_upsert + + await asyncio.gather( + trigger_decision_clustering( + _event(owner_id=owner_a, decision_entry_id="dc_run_a"), + FakeStrategyContext(), + ), + trigger_decision_clustering( + _event(owner_id=owner_b, decision_entry_id="dc_run_b"), + FakeStrategyContext(), + ), + ) + return log + + +async def test_partition_lock_serialises_runs_on_same_owner( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Two runs sharing ``owner_id`` must not overlap critical sections.""" + log = await _run_serialisation_probe("u_alice", "u_alice", monkeypatch) + assert log in ( + ["enter:dc_run_a", "leave:dc_run_a", "enter:dc_run_b", "leave:dc_run_b"], + ["enter:dc_run_b", "leave:dc_run_b", "enter:dc_run_a", "leave:dc_run_a"], + ) + + +async def test_partition_lock_lets_different_owners_run_in_parallel( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Runs on distinct ``owner_id`` must overlap (no false serialisation).""" + log = await _run_serialisation_probe("u_alice", "u_bob", monkeypatch) + assert log.index("enter:dc_run_a") < log.index("leave:dc_run_b") + assert log.index("enter:dc_run_b") < log.index("leave:dc_run_a") + + +@pytest.mark.asyncio +async def test_returns_without_side_effects_when_embedding_unavailable() -> None: + """Capability unavailable → early return; no embed, no repo, no emit.""" + ctx = FakeStrategyContext() + with ( + patch( + "everos.memory.strategies.trigger_decision_clustering" + ".get_embedding_capability", + return_value=EmbeddingCapability(provider=None), + ), + patch( + "everos.memory.strategies.trigger_decision_clustering.cluster_repo" + ) as mock_repo, + structlog.testing.capture_logs() as captured, + ): + mock_repo.list_for_owner = AsyncMock( + side_effect=AssertionError("cluster_repo must not be touched"), + ) + mock_repo.upsert_with_members = AsyncMock( + side_effect=AssertionError("cluster_repo must not be touched"), + ) + + await trigger_decision_clustering(_event(), ctx) + + assert ctx.emitted == [] + gated = [ + e + for e in captured + if e.get("event") == "strategy_gated_off_embedding_unavailable" + ] + assert len(gated) == 1 + assert gated[0]["strategy_name"] == "trigger_decision_clustering" + assert gated[0]["owner_id"] == "u_alice" + + +async def test_applies_to_rejects_non_pipeline_source() -> None: + """Events with source != 'pipeline' must not pass the applies_to gate.""" + meta = trigger_decision_clustering.meta + pipeline_event = _event() + assert meta.applies_to(pipeline_event) is True + + reflection_event = _event( + memcell_id="mc_merged", + decision_entry_id="dc_20260517_0002", + decision_text="merged decision", + source="reflection", + ) + assert meta.applies_to(reflection_event) is False diff --git a/tests/unit/test_service/test_memorize_engine_gate.py b/tests/unit/test_service/test_memorize_engine_gate.py index fb1349fd9..d838ac1ea 100644 --- a/tests/unit/test_service/test_memorize_engine_gate.py +++ b/tests/unit/test_service/test_memorize_engine_gate.py @@ -1,9 +1,10 @@ """Verify OME strategy registration is unconditional (capability check is body-guard). ``_get_engine()`` builds the singleton :class:`OfflineEngine` and registers -every OME strategy exactly once. Four strategies re-embed or depend on a +every OME strategy exactly once. Six strategies re-embed or depend on a cluster produced by re-embedding (``trigger_profile_clustering``, -``trigger_skill_clustering``, ``extract_agent_skill``, ``reflect_episodes``); +``trigger_decision_clustering``, ``trigger_skill_clustering``, +``extract_agent_skill``, ``reflect_episodes``, ``reflect_decisions``); these are now registered regardless of embed availability. Each guards its own body via :func:`get_embedding_capability` at execution time so a runtime tier upgrade (Tier 1 → Tier 2) picks up on the next dispatch @@ -24,15 +25,19 @@ _ALWAYS = { "extract_atomic_facts", + "extract_decision", "extract_foresight", "extract_agent_case", "extract_user_profile", + "extract_principles", } _REQUIRE_EMBED = { "trigger_profile_clustering", + "trigger_decision_clustering", "trigger_skill_clustering", "extract_agent_skill", "reflect_episodes", + "reflect_decisions", } @@ -70,7 +75,7 @@ def _registered_names() -> set[str]: def test_all_strategies_registered_when_embed_available( monkeypatch: pytest.MonkeyPatch, ) -> None: - """Embed available → all 8 strategies register.""" + """Embed available → all 12 strategies register.""" _set_embed_available(monkeypatch, available=True) names = _registered_names() assert names >= _ALWAYS | _REQUIRE_EMBED @@ -81,7 +86,7 @@ def test_all_strategies_registered_when_embed_unavailable( ) -> None: """Embed unavailable → registration is still unconditional. - The four embed-requiring strategies stay in the registry so that a + The six embed-requiring strategies stay in the registry so that a runtime tier upgrade (edit everos.toml + reload settings) takes effect on the next dispatch without a server restart. Each strategy body guards on :func:`get_embedding_capability` and no-ops when diff --git a/uv.lock b/uv.lock index ec09b1abb..40c2385a7 100644 --- a/uv.lock +++ b/uv.lock @@ -443,7 +443,7 @@ wheels = [ [[package]] name = "everalgo-agent-memory" -version = "0.4.0" +version = "0.5.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "asgiref" }, @@ -451,22 +451,24 @@ dependencies = [ { name = "everalgo-clustering" }, { name = "everalgo-core" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/a9/d1/c8f4eba92d3d9fbdcbfae9eccf887650a22ae92b16a5b791b0feaf07f8dc/everalgo_agent_memory-0.4.0.tar.gz", hash = "sha256:58364d4c78b39b91d1eb93489c7d37fa9343e696987ac0c21631a8bb1099ec57", size = 86143, upload-time = "2026-07-30T07:10:30.002Z" } +sdist = { url = "https://files.pythonhosted.org/packages/e7/95/efef646284bfcb8a078734c842d5f0175667b67d9728648903baf74e7e27/everalgo_agent_memory-0.5.0.tar.gz", hash = "sha256:b9edc8d7982330b183a3716da61ef7ec90ab5ca4125ceda8f6fcd7420b5e475d", size = 86741, upload-time = "2026-08-19T07:18:21.244Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/5e/7c/98632538a43a1b9a14310e95dd3400c1768b0a15b89072a49ce87e06ba46/everalgo_agent_memory-0.4.0-py3-none-any.whl", hash = "sha256:9b11d066a620df8cc3e6d2fa6cd088c44d9aa32eef30bc55b3c8e792efee6b4b", size = 62380, upload-time = "2026-07-30T07:10:29.105Z" }, + { url = "https://files.pythonhosted.org/packages/22/bc/512b0f149eccd806ac6f6a4dff763f9843edb09f0ce4c68723e893c486aa/everalgo_agent_memory-0.5.0-py3-none-any.whl", hash = "sha256:366867fc29971538ff3924a061a32b53bdbc96939e95339acdfb5b65481859eb", size = 62490, upload-time = "2026-08-19T07:18:20.3Z" }, ] [[package]] name = "everalgo-boundary" -version = "0.2.1" -source = { registry = "https://pypi.org/simple" } +version = "0.3.0" +source = { editable = "../EverAlgo/packages/everalgo-boundary" } dependencies = [ { name = "asgiref" }, { name = "everalgo-core" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/49/6e/3c60e7c253948e42906108bff3f8227733804801c3ea96e04f7895366686/everalgo_boundary-0.2.1.tar.gz", hash = "sha256:23a93bb36b06251e5a85765f68640c55bcfe0f1faf8b025e44ef8857a5ce36f9", size = 21946, upload-time = "2026-06-15T06:14:36.962Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/73/6f/6cb00d36ee007360ac0e2e62c44da782ef881f0f5def9f9d5a64142d88fb/everalgo_boundary-0.2.1-py3-none-any.whl", hash = "sha256:95ac8982291041b5641b13c915790ef20cca8c44018c2001ff608b13c7ae6b8d", size = 17192, upload-time = "2026-06-15T06:14:35.921Z" }, + +[package.metadata] +requires-dist = [ + { name = "asgiref", specifier = ">=3.0" }, + { name = "everalgo-core", editable = "../EverAlgo/packages/everalgo-core" }, ] [[package]] @@ -484,16 +486,19 @@ wheels = [ [[package]] name = "everalgo-core" -version = "0.3.0" -source = { registry = "https://pypi.org/simple" } +version = "0.6.0" +source = { editable = "../EverAlgo/packages/everalgo-core" } dependencies = [ { name = "openai" }, { name = "pydantic" }, { name = "tiktoken" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/53/d7/395c0504b426bf25cd9af60d5bab8f9c3131a6dee53a16b2a5a6a477b6c8/everalgo_core-0.3.0.tar.gz", hash = "sha256:cd91204a336ad459ae1c03eda97cdb0575534b675523cb775188debacc8f241b", size = 54469, upload-time = "2026-06-16T06:30:06.801Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/2b/5d/ad60747004d873b23443412a74449b39640f08c4f6548fd4442641c3ff17/everalgo_core-0.3.0-py3-none-any.whl", hash = "sha256:5a25b784a2d24e28a762fee5751174a68c2eda168c353c33740ca3183069c02b", size = 44118, upload-time = "2026-06-16T06:30:05.678Z" }, + +[package.metadata] +requires-dist = [ + { name = "openai", specifier = ">=1.0" }, + { name = "pydantic", specifier = ">=2.7" }, + { name = "tiktoken", specifier = ">=0.7" }, ] [[package]] @@ -547,17 +552,21 @@ wheels = [ [[package]] name = "everalgo-user-memory" -version = "0.4.0" -source = { registry = "https://pypi.org/simple" } +version = "0.8.0" +source = { editable = "../EverAlgo/packages/everalgo-user-memory" } dependencies = [ { name = "asgiref" }, { name = "everalgo-boundary" }, { name = "everalgo-core" }, { name = "pydantic" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/50/dc/e28a9901c13b5399ebba2033fca4e152206528d20b1b8168d53ccf6a6da0/everalgo_user_memory-0.4.0.tar.gz", hash = "sha256:68dca5d49a8586caa734f862ae64ad63178f32c6b9d10341ba69c0a2f58b43b9", size = 74997, upload-time = "2026-07-30T07:02:34.204Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/cf/ba/5c7e65281efae449b01b92f37181c39445f763a31b32f798da740413f17a/everalgo_user_memory-0.4.0-py3-none-any.whl", hash = "sha256:a40cc3ba8aa8062942572ed50e3434887fbb9e73e31d31885124506534d9dc3e", size = 65307, upload-time = "2026-07-30T07:02:33.263Z" }, + +[package.metadata] +requires-dist = [ + { name = "asgiref", specifier = ">=3.0" }, + { name = "everalgo-boundary", editable = "../EverAlgo/packages/everalgo-boundary" }, + { name = "everalgo-core", editable = "../EverAlgo/packages/everalgo-core" }, + { name = "pydantic", specifier = ">=2.7" }, ] [[package]] @@ -625,11 +634,11 @@ requires-dist = [ { name = "anyio", specifier = ">=4.0" }, { name = "apscheduler", specifier = ">=3.10.4,<4.0" }, { name = "click", specifier = ">=8.1" }, - { name = "everalgo-agent-memory", specifier = "==0.4.0" }, + { name = "everalgo-agent-memory", specifier = "==0.5.0" }, { name = "everalgo-knowledge", specifier = "==0.1.1" }, { name = "everalgo-parser", extras = ["svg"], marker = "extra == 'multimodal'", specifier = ">=0.2.1" }, { name = "everalgo-rank", specifier = "==0.4.1" }, - { name = "everalgo-user-memory", specifier = "==0.4.0" }, + { name = "everalgo-user-memory", editable = "../EverAlgo/packages/everalgo-user-memory" }, { name = "fastapi", specifier = ">=0.104.0" }, { name = "greenlet", specifier = ">=3.0" }, { name = "jieba", specifier = ">=0.42.1,<1.0" },