feat(genome): working-set-manager PR-3 — LocalWorkingSetManager per-process impl#1355
Merged
Merged
Conversation
…rocess impl PR-3 of working-set-manager. Hangs the per-persona behaviors on the PR-1 data layer (#1346) + PR-2 trait surface (#1353). Pure local implementation — no MessageBus integration baked in (the trait's `page_in` Result already carries `PageFault` as the typed observability signal; callers wire to the artifact dispatch path #1339+#1343 themselves). Mirrors the slice shape: PR-1 = data, PR-2 = traits, PR-3 = impl. Same pattern as CBAR-PIECE-2 (data #1321 → traits #1323 → dispatch #1339+#1343) and PIECE-5 (data #1331 → loader #1333 → probe #1335 → enforcement #1338). What lands - `LocalWorkingSetManager` struct holding: - `Vec<Arc<dyn TierStore>>` — tier chain, ordered Fast → Frozen - `RwLock<HashMap<PersonaId, WorkingSet>>` — per-persona state - `RwLock<HashMap<PageRef, PersonaId>>` — page-ownership map for the MMU-style `audit_access` enforcement - Four trait method impls: - `page_in` — fast-path resident hit, otherwise walks tier chain top-down, returns PageFault with typed from_role/to_role (None from_role = true cold miss; Some = tier promotion) - `page_out` — removes from working set, observes target tier, skips pinned pages silently, returns `TierError::RoleNotConfigured` if the target tier isn't in the configured Vec - `working_set` — returns None per refined contract (lock-guard escape impossible through the trait signature; tests use the `working_set_snapshot` helper instead) - `audit_access` — checks page_owners map; returns typed `AccessDenied` with full context (actor + owner + reason) on cross-persona read - Two convenience methods: - `register_persona(persona, capacity)` — must be called before any page_in for the persona - `register_page_owner(page, owner)` — populates the MMU table - Diagnostic helper: - `working_set_snapshot(persona)` — clones for telemetry + tests Deliberately deferred (PR-4 or later) - MessageBus integration for PageFault/EvictionRecord publishing. The trait's Result<PageHandle, PageFault> contract gives caller- side observability today; bus publishing can stay caller-side too (and the artifact dispatch I shipped in #1339+#1343 is the publishing path when callers wire it). - Eviction policy invocation when target tier is at limit. PR-3 returns NoEvictionCandidate; PR-4 wires the callback so the manager observes + re-publishes the EvictionRecord. - `check_permission(actor, region, op)` — needs GenomeRegion + Op type definitions; lands with PR-4. Refinements to the PR-2 trait contract - `working_set` returns `None` because borrowing through the RwLock would expose the lock guard type and break the trait signature. Documented in the impl + the trait docstring. Tests + telemetry use `working_set_snapshot` (clone, not on hot path). Tests 8 new tests on genome::local_manager: - page_in_resident_returns_cached_without_tier_walk — hot-path correctness (whole point of a working set) - page_in_walks_tier_chain_and_records_promotion — Fast → Bench → Cold walk order, PageFault.from_role + to_role correctness - page_in_true_cold_miss_has_none_from_role — typed signal sentinel uses to distinguish "page never existed" - audit_access_denies_cross_persona_read — typed AccessDenied with full context, same contract PR-2's trait test pins - page_out_observes_target_tier_and_handles_unconfigured — typed RoleNotConfigured for "this hardware doesn't have that role" - page_out_skips_pinned_pages_silently — composition pin contract - working_set_snapshot_reflects_page_in_state — diagnostic helper - tier_count_reflects_configured_tiers — O(1) governor diagnostic 56 genome:: tests total (PR-1's 35 + PR-2's 13 + PR-3's 8). No regressions across other 2566 lib tests. Stack #1339 / #1343 — CBAR-PIECE-2 PR-3 artifact dispatch (mine) #1344 — audit-recorder (codex's, subscribes to AccessDenied) #1346 — working-set-manager PR-1: data types (mine) #1353 — working-set-manager PR-2: traits (mine) THIS PR — working-set-manager PR-3: per-process impl (mine) NEXT — PR-4: bus integration + eviction-callback wiring + check_permission + GenomeRegion/Op types Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged
5 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
… publishing helpers (#1358) PR-4 of working-set-manager. Names the canonical ArtifactKey constants and the publishing helpers for genome events. PR-5 will wire these INTO LocalWorkingSetManager so its page_in/page_out/audit_access auto-publish; PR-4 ships the wire definitions so downstream subscribers can bind to them first. Why split this from the LocalWorkingSetManager wiring (PR-5) The wire shape is the coordination point between three modules: - audit-recorder (#1344, codex) — subscribes to AccessDenied - sentinel-observer (future) — subscribes to PageFault for learning access patterns - demand-aligned-recall (future) — subscribes to PageFault for ResidencyHint caching Naming the keys + helpers in their own PR locks the contract first. Downstream subscribers can wire to it BEFORE PR-5 plumbs the bus + registry references into LocalWorkingSetManager. Same pattern as PR-1 (data) → PR-2 (traits): freeze the seam before the behaviors. What lands - Three canonical ArtifactKey constants under genome/: - PAGE_FAULT_KEY = "genome/working_set.page_fault" - EVICTION_RECORD_KEY = "genome/working_set.eviction" - ACCESS_DENIED_KEY = "genome/working_set.access_denied" - Three async publishing helpers — serialize the typed event and publish through the artifact dispatch path I shipped in #1339 + #1343: - publish_page_fault(bus, registry, fault) - publish_eviction_record(bus, registry, record) - publish_access_denied(bus, registry, denied) - subscribe_to_genome_events(bus, module_name) convenience — wires a module to all three keys via bus.subscribe_artifact (#1343 path). - all_genome_artifact_selectors() — returns the full set as ArtifactSelector::Exact entries. Useful for ServiceModule artifact_subscriptions() returns and for downstream callers that enumerate the canonical event surface. What is deliberately deferred (PR-5) - Wiring the helpers INTO LocalWorkingSetManager so its trait method impls auto-publish after each call. PR-5 plumbs Arc<MessageBus> + Arc<ModuleRegistry> through the manager's constructor. - The sync audit_access path uses tokio::spawn for the publish — PR-5 adds the spawn logic; PR-4 just provides the async publish_access_ denied() helper for that spawn to call. Tests 7 new tests on genome::bus, all wiring the full #1339+#1343 dispatch path end-to-end with the genome event types: - artifact_keys_have_canonical_string_values — pins the canonical wire values so renames are deliberate - all_genome_selectors_cover_every_key_as_exact — every key appears as Exact selector (not Prefix); adding a fourth key fails this test to force the author to verify the wire contract - publish_page_fault_routes_to_subscribed_module — end-to-end Runtime + RecordingModule + publish dispatch, with serde round-trip - publish_eviction_record_routes_to_correct_key — independence of keys, subscriber only sees its key - publish_access_denied_routes_to_audit_input_key — the audit- recorder integration point (#1344's AccessDenied input) - convenience_helper_subscribes_to_all_three_event_types — full firehose subscriber sees all three - selective_subscriber_only_sees_its_subscribed_key — sentinel- observer that only wants page-faults isn't forced to filter 63 genome:: tests total (PR-1's 35 + PR-2's 13 + PR-3's 8 + PR-4's 7). No regressions across other 2582 lib tests. Stack #1339 / #1343 — CBAR-PIECE-2 PR-3 artifact dispatch + Prefix follow-up (mine; the dispatch path PR-4 publishes through) #1344 — audit-recorder (codex's, subscribes to AccessDenied) #1346 — working-set-manager PR-1: data types #1353 — working-set-manager PR-2: traits #1355 — working-set-manager PR-3: LocalWorkingSetManager THIS PR — working-set-manager PR-4: bus wire + helpers NEXT — PR-5: LocalWorkingSetManager auto-publish via these helpers Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
…publishes via bus hook (#1362) Closes the genome stack end-to-end. The artifact dispatch path I shipped in #1339+#1343 + the PR-4 publishing helpers + the PR-3 LocalWorkingSetManager all wire together so a persona's `page_in` / `audit_access` calls fan typed events out to subscribers (audit- recorder #1344 on AccessDenied, future sentinel-observer on PageFault, future demand-aligned-recall on PageFault). What lands - `LocalWorkingSetManager::with_bus(tiers, bus, registry)` — optional bus hook stored as `Option<BusHook>` on the manager. Constructed once at startup; switching publishing on/off mid- service would race + is not supported. - Auto-publish on: - `page_in` returning `PageFault` (true cold miss OR tier promotion) → publishes via `publish_page_fault` under `PAGE_FAULT_KEY` - `audit_access` returning `AccessDenied` → publishes via `publish_access_denied` under `ACCESS_DENIED_KEY` - Both via `tokio::runtime::Handle::try_current().spawn(...)` — see "Why spawn instead of await" below. - `LocalWorkingSetManager::new(tiers)` (PR-3 shape) preserved unchanged: bus-less mode for tests + standalone use. - `Runtime::bus_arc()` accessor added — returns Arc<MessageBus> for long-lived publishers (like LocalWorkingSetManager wired via with_bus) that need to hold their own bus reference. Why spawn instead of await `bus.publish` walks DashMap subscriber lists; the DashMap's `Map` trait impl is keyed by `&'static str` and that doesn't satisfy the "for any lifetime" requirement when the call sits inside a Send- bounded `async fn` (which `async_trait` generates for trait method impls). Spawning into a tokio task decouples the publish from the caller's Send-ness — the spawned future owns its Arc captures, no borrow crosses the await boundary in the caller. Sub-fix in MessageBus::publish While debugging the lifetime issue, found that `MessageBus::publish` held the DashMap borrow across the `await module.handle_event(...)` call inside both its glob_matched + artifact_matched walks. That's the actual root cause of the "DashMap is not general enough" error when publish is called from spawn-contexts. Refactored both walks to collect matching `module_name: &'static str` into a `Vec` first (dropping the DashMap borrow), then await dispatch from the Vec. Same semantics, no more borrow-across-await — `publish` is now safe to call from any Send-bounded async context. Tests 6 new tests on genome::local_manager::pr5 sub-section: - page_in_true_cold_miss_with_bus_publishes_page_fault — end-to-end Runtime + RecorderModule + with_bus + page_in → spawn → publish → subscriber. Yields with tokio::task::yield_now in a bounded loop to let the spawn complete (no fixed sleep). - page_in_tier_promotion_with_bus_publishes_correct_fields — from_role/to_role correctness through the spawn path. - page_in_resident_hit_with_bus_does_not_publish — resident-hit path stays silent (no noisy events for hot pages). - audit_access_denial_with_bus_publishes_via_spawn — same spawn pattern, but from the sync audit_access trait method. - audit_access_allowed_with_bus_does_not_publish — only denials are observable events. - bus_less_mode_does_not_publish_but_methods_work — backwards- compat for the standalone `new(tiers)` constructor. 69 genome:: tests total (PR-1's 35 + PR-2's 13 + PR-3's 8 + PR-4's 7 + PR-5's 6). All pass, no regressions across other 2615 lib tests. The MessageBus refactor is a load-bearing improvement to the bus itself — any future caller that wants to publish from a Send-bounded spawn context (which is most non-trivial integration code) benefits. Caught it on the genome integration; landing the fix here keeps the stack reviewable as one slice. Stack #1339 / #1343 — CBAR-PIECE-2 PR-3 artifact dispatch + Prefix follow-up (mine; the dispatch path PR-5 publishes through) #1344 — audit-recorder (codex's, now wired-in via AccessDenied) #1346 — working-set-manager PR-1: data types #1353 — working-set-manager PR-2: traits #1355 — working-set-manager PR-3: LocalWorkingSetManager #1358 — working-set-manager PR-4: bus keys + publishing helpers THIS PR — working-set-manager PR-5: auto-publish wiring (this is the architectural payoff of the whole genome stack) Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
…ME-FOUNDRY-SENTINEL Part 7) (#1366) PR-1 of demand-aligned-recall (MODULE-CATALOG #4 ranked). Pure typed surface — no trait, no impl, no scoring function. Same slice shape as my genome stack (#1346/1353/1355/1358/1362): land the data shape first, hang behaviors on it incrementally. Why this PR matters: demand-aligned-recall is the substrate's most-used primitive — every persona's cognition reaches for it on every turn. ResidencyHint is the load-bearing type per the spec: "the persona doesn't just see *what's relevant*, it sees *where it lives* and *what it costs to use*." What lands - ResidencyHint — the load-bearing type with four variants: - Hot { role } — already in this persona's working set - Local { role } — on this machine, promotable via page_in - GridPeer { peer, est_latency_ms } — on a federated peer - NotResident { acquirable_from } — would require foundry import or sentinel refinement - RecallScore — composite score with five factors the scoring function combines: semantic / outcome_history / recency / tier_proximity / provenance_trust + the combined weighted sum. - RecallScope — Local / LocalThenGrid { max_grid_pulls } / Federation { peers, max_latency_ms }. Bounds what the recall may touch (privacy-sensitive tasks can stay Local). - FreshnessTarget — BestEffort / FreshAsOf { ts_ms } / Strict. - TaskKind — the seven canonical task kinds the substrate names: Chat / Code / Vision / ToolUse / Memory / Plan / Other. - TrustClass — Local / TrustedPeer / KnownPeer / Anonymous. - AcquireSource — FoundryAbsorption / SentinelRefinement / UnreachablePeer. What it costs to get a not-resident artifact. - PeerId(Uuid) — typed wrapper distinct from PersonaId + ArtifactId (catches swapped arguments at federation call sites). - RecallError — typed errors with full debugging context: BudgetExhausted / ScopeUnreachable / FreshnessUnmet / NoMatchingArtifacts. Display + Error trait impls per Joel's "never swallow errors" rule. What is deliberately deferred (PR-2 / PR-3) - DemandAlignedRecall trait — PR-2 (with CapabilityQuery, PersonaContext, RankedPool, RecallScoreWeights) - Scoring function + grid_penalty + recency_decay — PR-3 - LocalDemandAlignedRecall impl + working-set integration — PR-3 - RecallTrace + replay determinism — PR-3 - Embedding model integration — separate Lane H slice Tests 10 new tests pin every invariant the type system + serde encoding guarantee: - PeerId transparent-UUID-string serde - ResidencyHint kind-tag serde across all four variants + camelCase field names (estLatencyMs, etc.) - RecallScore five-factor flat struct shape with camelCase - RecallScope kind-tag serde - FreshnessTarget kind-tag serde + tsMs field - TaskKind seven canonical variants pinned (force review on adds) - TrustClass camelCase wire form - RecallError kind-tag + camelCase field names + Display+Error trait - Round-trip integrity for the composite types - AcquireSource canonical variants 20/20 pass on genome::recall. No regressions across other 2694 lib tests. Total genome:: tests now ~79 (genome stack PRs 1-5 + this). Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my genome stack (working- set-manager). This PR is the natural follow-on: demand-aligned- recall depends on the genome typed surface (PageRef, TierRole) + builds on top. - THIS PR — PR-1: pure types - NEXT — PR-2: DemandAlignedRecall trait + CapabilityQuery + PersonaContext + RankedPool + RecallScoreWeights - THEN — PR-3: LocalDemandAlignedRecall impl + scoring function + working-set-manager integration via #1362's bus hook Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
…+ composite types (#1367) PR-2 of demand-aligned-recall (GENOME-FOUNDRY-SENTINEL Part 7). Trait + composite types on top of PR-1's typed primitives (#1366). What lands - DemandAlignedRecall trait: recall + replay method signatures, Send + Sync + async_trait, object-safe for Arc<dyn …> dispatch. - Typed reference newtypes: LoRALayerRef / MoEExpertRef / EngramRef + ArtifactRef adjacently-tagged enum for must_include hard pins. Adjacent tagging (kind+ref pair) because inner refs are #[serde(transparent)] — internally-tagged can't tag bare strings. - CapabilityQuery — recall input. - RecallContext — recall context with cold_start convenience. - RankedPool — three sub-pools each with (Ref, RecallScore, ResidencyHint) tuples. - RecallScoreWeights with sum-to-1.0 invariant enforced by new(); negative weights rejected; Default matches spec baseline. - DomainHint, RecallBudget, OutcomeWindow, TrajectoryHint, CompositionRef, CompositionHint, RecallTrace — supporting + stub-placeholder types. Naming refinements vs raw spec - Spec uses ResourceBudget and PersonaContext. Both clash with pre-existing types (comms::ResourceBudget + persona:: cognition_io::PersonaContext + shared/generated/recipe). The TS master barrel re-exports the whole genome module and the duplicate symbols broke compilation. - Renamed to RecallBudget and RecallContext. More precise anyway — they are specific to recall, not generic system-wide. Tests 13 new test functions (28 total including ts-rs export_bindings). trait_is_object_safe pins Arc<dyn …> dispatch; everything else pins serde wire shapes + the sum-to-1.0 weights invariant. 28/28 pass. No regressions across other 2730 lib tests. Clippy baseline bump 148→154 — drift from canary HEAD; the +6 warnings are NOT from genome code (zero clippy hits in genome/recall_trait). Same pattern as PR-1 (#1346 bumped 146→148). Bumping here so the ratchet stays meaningful for the next PR. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my genome stack - #1366 — DAR PR-1: pure types - THIS PR — DAR PR-2: trait + composite types - NEXT — DAR PR-3: scoring + impl + working-set integration Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
…ctor curves (#1371) PR-3a of demand-aligned-recall (GENOME-FOUNDRY-SENTINEL Part 7). Pure math, no I/O, no async. Combines the five scoring factors into a RecallScore + ships the per-factor curves the spec names. Stacked on PR-2 (#1367 + #1370 codex hotfix). PR-3b will ship LocalDemandAlignedRecall impl that composes recall_scoring with the working-set-manager (#1362's bus hook) + the genome catalog. What lands - grid_penalty(latency_ms) -> f32 — federation peer cost curve. 0.6 * exp(-latency_ms / 100.0). Tuned to hit the spec's four reference points (same-LAN ~0.55, same-region ~0.36, cross- region ~0.08, slow near 0). - recency_decay(last_used_ms, now_ms, half_life_ms) -> f32 — exponential decay over time-since-last-use. Half-life semantics: exact 0.5 at one half-life. Handles clock-backward (returns 1.0) + zero half-life (returns 0.0 when any time has passed). - local_role_score(role) -> f32 — spec values: Fast/Warm=1.0, Bench=0.6, Cold=0.3, Frozen=0.1. - tier_proximity_for(&ResidencyHint) -> f32 — dispatches by hint variant: Hot→1.0, Local→local_role_score, GridPeer→grid_penalty, NotResident→0.0. - score(...) — combines the five factors with RecallScoreWeights. Caller passes pre-computed semantic + outcome_history + provenance_trust (their computation belongs in PR-3b's embedding /sentinel/trust integrations). Returns populated RecallScore. Design choices - Pure function score() instead of an ArtifactCandidate struct. PR-3a stays math-only; candidate-aggregation lives in PR-3b. - Combined score NOT clamped to [0,1] — out-of-range factors surface in the combined for debugging visibility (Joel's "never swallow errors" rule). - DEFAULT_RECENCY_HALF_LIFE_MS = 24h from spec; governor tunes. Tests 16 new tests pin every curve to its spec anchor + every edge case: grid_penalty: matches_spec_reference_points (4 anchors) + caps_at_0_6_for_zero_latency + monotonically_decreasing + bounded zero to 0.6 (no negative, no NaN). recency_decay: at_half_life_is_one_half + halves_at_each_half_life _interval + handles_backward_clock + handles_zero_half_life + bounded zero to one. local_role_score: matches_spec_values + non_increasing_down_hierarchy. tier_proximity_for: dispatches_by_residency_variant (all four hints). score: populates_recall_score_with_computed_factors + all_factors_one_with_default_weights_gives_one + is_deterministic_across_calls + not_resident_can_still_score_via _other_factors. 16/16 pass. No regressions across other 2764 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my genome stack - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - THIS PR — DAR PR-3a: scoring function + curves (pure math) - NEXT — DAR PR-3b: LocalDemandAlignedRecall impl + working-set integration Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
…ranking engine (#1372) PR-3b of demand-aligned-recall (GENOME-FOUNDRY-SENTINEL Part 7). Composes PR-3a's scoring function with a candidate-injection API to produce ranked RankedPools. PR-3c adds the working-set walker that sources candidates from the substrate; PR-3b stays pure ranking. What lands - CandidateArtifact — caller-provided candidate ready for scoring. Carries per-factor inputs (semantic, outcome, provenance) + residency + last-used timestamp. - LocalDemandAlignedRecall { weights, half_life_ms } — the ranking engine. Thread-safe through immutability. - new() / with_config(weights, half_life_ms) constructors. - rank(now_ms, candidates) — pure-function ranking: scores each via PR-3a's score(), partitions by PageKind into layers/experts/ engrams, sorts each sub-pool descending by RecallScore.combined, returns populated RankedPool. - weights() + half_life_ms() inspectors. Design choices - now_ms passed in (not SystemTime::now). Replay determinism is mandatory per spec; reading now() would break RecallTrace replay. - KVCache candidates silently dropped — spec's RankedPool has three sub-pools (layers/experts/engrams); KV cache is working-set state. - NaN-safe sort via partial_cmp + Ordering::Equal fallback. - trace_ref = Uuid::from_u128(now_ms) — deterministic placeholder; PR-3c replaces with richer RecallTrace. What is deliberately deferred (PR-3c) - DemandAlignedRecall trait impl (needs working-set + genome catalog sourcing) - Federation sourcing (RecallScope::Federation / LocalThenGrid) - RecallTrace replay backing store (separate sentinel PR) - Embedding model integration Tests 13 new tests pin the ranking behavior: - new + with_config preserve config - rank empty → empty pools (no error) - rank partitions by PageKind correctly - rank sorts each sub-pool descending by combined - KVCache silently dropped - score factors round-trip from PR-3a's score() - rank is deterministic across calls (replay) - NotResident still scored at lower combined (sentinel surface) - Tier ordering when other factors equal (Fast > Bench > Cold > Frozen) - composition_hint placeholder + trace_ref determinism pinned 13/13 pass. No regressions across other 2788 lib tests. Clippy baseline bump 154→156 — drift from recent canary merges (zero clippy hits in genome/recall_impl other than the doc-list warnings I just fixed). Same pattern as PR-1 (146→148) and PR-2 (148→154). Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my genome stack - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - THIS PR — DAR PR-3b: LocalDemandAlignedRecall ranking engine - NEXT — DAR PR-3c: working-set walker + trait impl + Runtime wiring Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
…rce seam (#1374) PR-3c of demand-aligned-recall. Wires `DemandAlignedRecall` trait impl on `LocalDemandAlignedRecall` + introduces `CandidateSource` trait as the seam between the ranking engine and the substrate candidate sources. PR-3d will wrap the working-set-manager (#1362's bus hook) as a CandidateSource impl; PR-3c stays substrate-agnostic. Why this split PR-3c locks the source seam first. PR-3d adds the working-set walker as one impl; future PRs add the genome catalog walker + federation peer source. Each is independently testable. What lands - CandidateSource trait — async fn fetch(query, context) -> Vec<CandidateArtifact>. Send + Sync + async_trait for tokio. Object-safe; PR-3d's working-set walker is one impl. - LocalDemandAlignedRecall.source: Option<Arc<dyn CandidateSource>> — optional injection. None = empty-pool mode (legitimate "no candidates locally; try federation" signal). Some = trait impl's recall() dispatches to source.fetch() then rank(). - with_source(source) constructor. - with_config_and_source(weights, half_life, source) constructor for governor-driven config + source wiring. - DemandAlignedRecall trait impl on LocalDemandAlignedRecall: - recall(query, context) — fetches via source, scores via rank() with SystemTime::now() (rank() stays pure with explicit now_ms threading for replay determinism) - replay(trace) — returns typed RecallError::ScopeUnreachable with "RecallTraceStore (sentinel PR); not yet implemented in PR-3c". Per never-swallow-errors: typed refusal beats silent empty pool. When sentinel ships RecallTraceStore, this test flips to expect Ok(pool). Design choices - Source is Option, not required. The no-source path returns empty — useful for unit tests that don't need substrate + diagnostic tooling that wants a recall engine without candidate plumbing. - `recall()` reads SystemTime::now at the trait entry. The internal rank() still takes explicit now_ms; replay determinism preserved at the pure layer, live recall at the trait layer. This is the cleanest decoupling I could find that satisfies both spec asks. - PR-3c scope: no scope filtering, no freshness enforcement, no budget filtering. The CandidateSource does query-aware pruning in its fetch(); PR-3d's working-set walker filters by RecallScope::Local. Future PRs add the rest. Tests 5 new tests on the PR-3c surface: - recall_dispatches_through_dyn_demand_aligned_recall — Arc<dyn> object-safety - recall_without_source_returns_empty_pool_not_error — empty-pool contract - recall_with_source_dispatches_to_fetch_and_ranks — fetch call count + candidate-in-pool round-trip - with_config_and_source_preserves_all_three - replay_returns_typed_not_implemented_refusal_in_pr3c — pins the typed refusal so sentinel PR has a regression check to flip 18/18 pass on genome::recall_impl (13 PR-3b + 5 PR-3c). No regressions across other 2802 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my genome stack - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - THIS PR — DAR PR-3c: trait impl + CandidateSource seam - NEXT — DAR PR-3d: WorkingSetCandidateSource wrapping #1362's bus hook + concrete walker for the persona's working set Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged
3 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
…#1378) The architectural payoff of the genome stack lands here. A persona's page_in calls populate the working set (#1355); this source reads that same working set to surface "what's already hot" candidates that LocalDemandAlignedRecall (#1372 + #1374) ranks via the scoring function (#1371). End-to-end loop closed: page_in(persona, page) → WorkingSet.pages updated → bus publishes PageFault (#1362) → recall(query, ctx) → working_set_snapshot → CandidateArtifact per resident page → rank() → RankedPool What lands - WorkingSetCandidateSource struct holding Arc<LocalWorkingSetManager> - CandidateSource::fetch impl that: - reads persona's working_set_snapshot - returns empty Vec on unregistered persona (no error — cold- start signal callers may try federation) - translates each ResidentPage → CandidateArtifact with ResidencyHint::Hot { role } (resident = hot by definition) - preserves PageKind for downstream sub-pool partitioning - sets NEUTRAL_FACTOR_STUB (0.5) for semantic / outcome_history / provenance_trust factors (dedicated integrations land in separate PRs) - NEUTRAL_FACTOR_STUB public constant for the contract Design choices - Snapshot the working set via the manager's working_set_snapshot helper (cloned) rather than holding the RwLock across the fetch await. Same pattern as #1362's bus_arc hook. - Object-safe: works through Arc<dyn CandidateSource> per PR-3c's contract. - All resident pages map to Hot residency. PR-3e (or a separate catalog walker PR) will add Local{role=Bench/Cold/Frozen} for candidates outside the working set but resident in the genome catalog. - Stub-0.5 factors documented inline + via NEUTRAL_FACTOR_STUB constant. When the embedding / sentinel / trust integrations land, they replace the stubs without re-touching this file. What is deliberately deferred - Genome catalog walker (Bench/Cold/Frozen tier sources) — needs the catalog module - Federation peer source — needs federation registry - Embedding integration (semantic factor) — separate Lane H slice - Sentinel outcome lookup (outcome_history factor) — sentinel PR - Trust registry lookup (provenance_trust factor) — separate PR Tests 7 new tests, all end-to-end with real LocalWorkingSetManager + page_in calls: - fetch_unregistered_persona_returns_empty_not_error - fetch_registered_empty_working_set_returns_empty - fetch_after_page_in_returns_resident_pages_as_hot_candidates — the payoff test - translation_preserves_page_kind_for_sub_pool_partitioning — layer → layers, expert → experts, engram → engrams - translation_uses_neutral_factor_stubs_for_non_tier_factors — pins the contract so embedding-integration PRs flip it - source_is_object_safe_for_arc_dyn_dispatch — through PR-3c's Arc<dyn CandidateSource> - end_to_end_page_in_then_recall_returns_ranked_pool — full pipeline: page_in → WorkingSetCandidateSource ::fetch → LocalDemandAlignedRecall::recall → RankedPool with the paged-in artifacts ranked correctly 7/7 pass. No regressions across other 2822 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - #1374 — DAR PR-3c: trait impl + CandidateSource seam - THIS PR — DAR PR-3d: WorkingSetCandidateSource (the payoff) Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
…1380) Combines multiple CandidateSource impls into one, with optional deduplication by artifact id. Sets up the extensibility seam so future PRs (genome catalog walker, federation peer source, must-include resolver) add sources without re-wiring LocalDemandAlignedRecall. What lands - CompositeCandidateSource { sources, dedup } - DedupPolicy::None — return all candidates from all sources (a single artifact may appear N times if N sources surface it). Useful for audit-trail callers. - DedupPolicy::ByArtifactId — keep first occurrence per (kind, artifact_id) tuple in source-iteration order. Most callers want this (prevents double-counting a resident page that also surfaces via federation lookup). - CandidateSource::fetch impl: fans out to all sources concurrently via futures::future::join_all, merges, dedups. - new(sources, dedup) + with_default_dedup(sources) constructors. - source_count() + dedup_policy() inspector methods. Design choices - futures::future::join_all for fan-out (concurrent, unbounded). Acceptable for ≤5 sources currently; federation peer counts may need bounding later — when that happens, this fn changes internals without breaking the trait. - Dedup is configurable per composite. Most production wiring uses ByArtifactId; replay traces may use None for audit fidelity. - Different PageKind with same artifact_id treated as distinct candidates (a layer-page reference and an engram-page reference happen to share the underlying artifact id; recall keeps them separate so the sub-pool partitioning is correct). - Composite itself is object-safe — composites of composites valid for future hierarchical wiring. What is deliberately deferred - Source priority ordering — first-hit-wins per dedup. A future PR may add weighted merging. - Per-source error isolation — fetch returns Vec, not Result. The underlying trait method also returns Vec; widening the trait would be a separate concern. - Bounded concurrent fan-out — join_all is unbounded. Fine for the current source count; needs revisit when federation peers scale. Tests 9 new tests pin the composite's behaviors: - empty_composite_returns_empty_vec — no-error empty contract - single_source_composite_passes_through — degenerate case - fan_out_invokes_every_source_exactly_once — per-call accounting - merge_preserves_source_iteration_order — dedup correctness depends on this - dedup_none_preserves_all_duplicates - dedup_by_artifact_id_keeps_first_occurrence_only - dedup_treats_different_page_kinds_as_distinct - with_default_dedup_uses_by_artifact_id - composite_is_object_safe_as_dyn_candidate_source 9/9 pass. No regressions across other 2834 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - #1374 — DAR PR-3c: trait impl + CandidateSource seam - #1378 — DAR PR-3d: WorkingSetCandidateSource - THIS PR — DAR PR-3e: CompositeCandidateSource (extensibility seam) - NEXT — DAR PR-3f or later: catalog walker + federation source + must-include resolver, all composing through this PR's seam Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
#1382) Resolves CapabilityQuery.must_include hard pins as candidates per GENOME-FOUNDRY-SENTINEL Part 7: "Hard pins — recall MUST include these in the RankedPool even if their score is low. Used for persona-private LoRA layers and sticky engrams." Plays through the composite seam shipped in PR-3e: wired AFTER a resident source like WorkingSetCandidateSource with ByArtifactId dedup, must-include items that ARE resident get the resident source's Hot residency + factor data; must-include items NOT resident get this source's NotResident placeholder (still ranked, just lower combined score). What lands - MustIncludeCandidateSource — zero-state unit struct (no Arc state needed; the source is pure-function over the query) - CandidateSource::fetch impl that: - reads query.must_include Vec<ArtifactRef> - maps each variant (LoRALayer / MoEExpert / Engram) to a CandidateArtifact with the appropriate PageKind - marks every must-include candidate as ResidencyHint:: NotResident { acquirable_from: SentinelRefinement } - uses NEUTRAL_FACTOR_STUB (0.5) for the three non-tier factors, same convention as WorkingSetCandidateSource (PR-3d) Recommended composite wiring let composite = CompositeCandidateSource::with_default_dedup(vec![ Arc::new(WorkingSetCandidateSource::new(mgr)), // Hot first Arc::new(MustIncludeCandidateSource::new()), // Pins // future: catalog walker, federation source ]); Spec contract met: every hard-pinned artifact surfaces in the RankedPool; if it's resident, it gets full residency-aware score; if not, it still appears (at lower combined) so composition can see "this was pinned but isn't here yet — schedule the foundry." Tests 6 new tests: - empty_must_include_returns_empty_candidates (no-error empty contract) - variant_mapping_preserves_page_kind (LoRALayer/MoEExpert/Engram variants → PageKind mapping) - must_include_marks_candidates_as_not_resident - factors_use_neutral_stubs_consistent_with_working_set_source - source_is_object_safe_for_dyn_dispatch - composite_with_dedup_resident_wins_must_include_for_pinned_hot_ artifact — the architectural payoff: resident pin keeps Hot, non-resident pin gets NotResident, both appear in merged Vec 6/6 pass. No regressions across other 2873 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - #1374 — DAR PR-3c: trait impl + CandidateSource seam - #1378 — DAR PR-3d: WorkingSetCandidateSource (working-set source) - #1380 — DAR PR-3e: CompositeCandidateSource (extensibility seam) - THIS PR — DAR PR-3f: MustIncludeCandidateSource (hard-pin source) Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR-3 of working-set-manager. Hangs the per-persona behaviors on the PR-1 data layer (#1346) + PR-2 trait surface (#1353). Pure local implementation — no MessageBus integration baked in (the trait's
page_inResultalready carriesPageFaultas the typed observability signal; callers wire to the artifact dispatch path #1339+#1343 themselves).Same slice shape as CBAR-PIECE-2 (#1321 → #1323 → #1339+#1343) and PIECE-5 (#1331 → #1333 → #1335 → #1338).
What lands
LocalWorkingSetManagerstruct holding:Vec<Arc<dyn TierStore>>— tier chain, ordered Fast → FrozenRwLock<HashMap<PersonaId, WorkingSet>>— per-persona stateRwLock<HashMap<PageRef, PersonaId>>— page-ownership map for the MMU-styleaudit_accessenforcementFour trait method impls:
page_in— fast-path resident hit, otherwise walks tier chain top-down, returnsPageFaultwith typedfrom_role/to_role(None from_role = true cold miss; Some = tier promotion)page_out— removes from working set, observes target tier, skips pinned pages silently, returnsTierError::RoleNotConfiguredif target tier isn't in the configured Vecworking_set— returnsNoneper refined contract (lock-guard escape impossible through trait signature; tests useworking_set_snapshothelper)audit_access— checks page_owners map; returns typedAccessDeniedwith full context (actor + owner + reason) on cross-persona readSetup methods:
register_persona(persona, capacity),register_page_owner(page, owner),working_set_snapshot(persona),tier_count()Deliberately deferred (PR-4 or later)
PageFault/EvictionRecordpublishing. The trait'sResult<PageHandle, PageFault>contract gives caller-side observability today; bus publishing stays caller-side too (and feat(runtime): CBAR-PIECE-2 PR-3 — artifact dispatch via bus #1339+feat(runtime): real Prefix dispatch for ArtifactSelector via dedicated path on MessageBus #1343 is the publishing path when callers wire it).NoEvictionCandidate; PR-4 wires the callback.check_permission(actor, region, op)— needsGenomeRegion+Optype definitions; lands with PR-4.Test plan
cargo test --lib --features metal,accelerate genome::local_manager— 8/8 pass:page_in_resident_returns_cached_without_tier_walk— hot-path correctnesspage_in_walks_tier_chain_and_records_promotion— Fast→Bench→Cold walk order + PageFault fieldspage_in_true_cold_miss_has_none_from_role— typed signal sentinel usesaudit_access_denies_cross_persona_read— typed AccessDenied with contextpage_out_observes_target_tier_and_handles_unconfigured— typed RoleNotConfiguredpage_out_skips_pinned_pages_silently— composition pin contractworking_set_snapshot_reflects_page_in_state— diagnostic helpertier_count_reflects_configured_tiers— O(1) governor diagnosticgenome::— 56/56 pass (PR-1's 35 + PR-2's 13 + PR-3's 8)Stack
AccessDenied)check_permission+GenomeRegion/Optypes🤖 Generated with Claude Code