feat(genome): demand-aligned-recall PR-3d — WorkingSetCandidateSource (genome stack payoff)#1378
Merged
joelteply merged 1 commit intoMay 17, 2026
Conversation
The architectural payoff of the genome stack lands here. A persona's page_in calls populate the working set (#1355); this source reads that same working set to surface "what's already hot" candidates that LocalDemandAlignedRecall (#1372 + #1374) ranks via the scoring function (#1371). End-to-end loop closed: page_in(persona, page) → WorkingSet.pages updated → bus publishes PageFault (#1362) → recall(query, ctx) → working_set_snapshot → CandidateArtifact per resident page → rank() → RankedPool What lands - WorkingSetCandidateSource struct holding Arc<LocalWorkingSetManager> - CandidateSource::fetch impl that: - reads persona's working_set_snapshot - returns empty Vec on unregistered persona (no error — cold- start signal callers may try federation) - translates each ResidentPage → CandidateArtifact with ResidencyHint::Hot { role } (resident = hot by definition) - preserves PageKind for downstream sub-pool partitioning - sets NEUTRAL_FACTOR_STUB (0.5) for semantic / outcome_history / provenance_trust factors (dedicated integrations land in separate PRs) - NEUTRAL_FACTOR_STUB public constant for the contract Design choices - Snapshot the working set via the manager's working_set_snapshot helper (cloned) rather than holding the RwLock across the fetch await. Same pattern as #1362's bus_arc hook. - Object-safe: works through Arc<dyn CandidateSource> per PR-3c's contract. - All resident pages map to Hot residency. PR-3e (or a separate catalog walker PR) will add Local{role=Bench/Cold/Frozen} for candidates outside the working set but resident in the genome catalog. - Stub-0.5 factors documented inline + via NEUTRAL_FACTOR_STUB constant. When the embedding / sentinel / trust integrations land, they replace the stubs without re-touching this file. What is deliberately deferred - Genome catalog walker (Bench/Cold/Frozen tier sources) — needs the catalog module - Federation peer source — needs federation registry - Embedding integration (semantic factor) — separate Lane H slice - Sentinel outcome lookup (outcome_history factor) — sentinel PR - Trust registry lookup (provenance_trust factor) — separate PR Tests 7 new tests, all end-to-end with real LocalWorkingSetManager + page_in calls: - fetch_unregistered_persona_returns_empty_not_error - fetch_registered_empty_working_set_returns_empty - fetch_after_page_in_returns_resident_pages_as_hot_candidates — the payoff test - translation_preserves_page_kind_for_sub_pool_partitioning — layer → layers, expert → experts, engram → engrams - translation_uses_neutral_factor_stubs_for_non_tier_factors — pins the contract so embedding-integration PRs flip it - source_is_object_safe_for_arc_dyn_dispatch — through PR-3c's Arc<dyn CandidateSource> - end_to_end_page_in_then_recall_returns_ranked_pool — full pipeline: page_in → WorkingSetCandidateSource ::fetch → LocalDemandAlignedRecall::recall → RankedPool with the paged-in artifacts ranked correctly 7/7 pass. No regressions across other 2822 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - #1374 — DAR PR-3c: trait impl + CandidateSource seam - THIS PR — DAR PR-3d: WorkingSetCandidateSource (the payoff) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
…1380) Combines multiple CandidateSource impls into one, with optional deduplication by artifact id. Sets up the extensibility seam so future PRs (genome catalog walker, federation peer source, must-include resolver) add sources without re-wiring LocalDemandAlignedRecall. What lands - CompositeCandidateSource { sources, dedup } - DedupPolicy::None — return all candidates from all sources (a single artifact may appear N times if N sources surface it). Useful for audit-trail callers. - DedupPolicy::ByArtifactId — keep first occurrence per (kind, artifact_id) tuple in source-iteration order. Most callers want this (prevents double-counting a resident page that also surfaces via federation lookup). - CandidateSource::fetch impl: fans out to all sources concurrently via futures::future::join_all, merges, dedups. - new(sources, dedup) + with_default_dedup(sources) constructors. - source_count() + dedup_policy() inspector methods. Design choices - futures::future::join_all for fan-out (concurrent, unbounded). Acceptable for ≤5 sources currently; federation peer counts may need bounding later — when that happens, this fn changes internals without breaking the trait. - Dedup is configurable per composite. Most production wiring uses ByArtifactId; replay traces may use None for audit fidelity. - Different PageKind with same artifact_id treated as distinct candidates (a layer-page reference and an engram-page reference happen to share the underlying artifact id; recall keeps them separate so the sub-pool partitioning is correct). - Composite itself is object-safe — composites of composites valid for future hierarchical wiring. What is deliberately deferred - Source priority ordering — first-hit-wins per dedup. A future PR may add weighted merging. - Per-source error isolation — fetch returns Vec, not Result. The underlying trait method also returns Vec; widening the trait would be a separate concern. - Bounded concurrent fan-out — join_all is unbounded. Fine for the current source count; needs revisit when federation peers scale. Tests 9 new tests pin the composite's behaviors: - empty_composite_returns_empty_vec — no-error empty contract - single_source_composite_passes_through — degenerate case - fan_out_invokes_every_source_exactly_once — per-call accounting - merge_preserves_source_iteration_order — dedup correctness depends on this - dedup_none_preserves_all_duplicates - dedup_by_artifact_id_keeps_first_occurrence_only - dedup_treats_different_page_kinds_as_distinct - with_default_dedup_uses_by_artifact_id - composite_is_object_safe_as_dyn_candidate_source 9/9 pass. No regressions across other 2834 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - #1374 — DAR PR-3c: trait impl + CandidateSource seam - #1378 — DAR PR-3d: WorkingSetCandidateSource - THIS PR — DAR PR-3e: CompositeCandidateSource (extensibility seam) - NEXT — DAR PR-3f or later: catalog walker + federation source + must-include resolver, all composing through this PR's seam Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
joelteply
added a commit
that referenced
this pull request
May 17, 2026
#1382) Resolves CapabilityQuery.must_include hard pins as candidates per GENOME-FOUNDRY-SENTINEL Part 7: "Hard pins — recall MUST include these in the RankedPool even if their score is low. Used for persona-private LoRA layers and sticky engrams." Plays through the composite seam shipped in PR-3e: wired AFTER a resident source like WorkingSetCandidateSource with ByArtifactId dedup, must-include items that ARE resident get the resident source's Hot residency + factor data; must-include items NOT resident get this source's NotResident placeholder (still ranked, just lower combined score). What lands - MustIncludeCandidateSource — zero-state unit struct (no Arc state needed; the source is pure-function over the query) - CandidateSource::fetch impl that: - reads query.must_include Vec<ArtifactRef> - maps each variant (LoRALayer / MoEExpert / Engram) to a CandidateArtifact with the appropriate PageKind - marks every must-include candidate as ResidencyHint:: NotResident { acquirable_from: SentinelRefinement } - uses NEUTRAL_FACTOR_STUB (0.5) for the three non-tier factors, same convention as WorkingSetCandidateSource (PR-3d) Recommended composite wiring let composite = CompositeCandidateSource::with_default_dedup(vec![ Arc::new(WorkingSetCandidateSource::new(mgr)), // Hot first Arc::new(MustIncludeCandidateSource::new()), // Pins // future: catalog walker, federation source ]); Spec contract met: every hard-pinned artifact surfaces in the RankedPool; if it's resident, it gets full residency-aware score; if not, it still appears (at lower combined) so composition can see "this was pinned but isn't here yet — schedule the foundry." Tests 6 new tests: - empty_must_include_returns_empty_candidates (no-error empty contract) - variant_mapping_preserves_page_kind (LoRALayer/MoEExpert/Engram variants → PageKind mapping) - must_include_marks_candidates_as_not_resident - factors_use_neutral_stubs_consistent_with_working_set_source - source_is_object_safe_for_dyn_dispatch - composite_with_dedup_resident_wins_must_include_for_pinned_hot_ artifact — the architectural payoff: resident pin keeps Hot, non-resident pin gets NotResident, both appear in merged Vec 6/6 pass. No regressions across other 2873 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - #1374 — DAR PR-3c: trait impl + CandidateSource seam - #1378 — DAR PR-3d: WorkingSetCandidateSource (working-set source) - #1380 — DAR PR-3e: CompositeCandidateSource (extensibility seam) - THIS PR — DAR PR-3f: MustIncludeCandidateSource (hard-pin source) Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The architectural payoff of the genome stack. A persona's
page_incalls populate the working set (my #1355); this source reads that same working set to surface "what's already hot" candidates thatLocalDemandAlignedRecall(#1372 + #1374) ranks via the scoring function (#1371).End-to-end loop now closed:
What lands
WorkingSetCandidateSourceholdingArc<LocalWorkingSetManager>CandidateSource::fetchimpl that:working_set_snapshotResidentPage→CandidateArtifactwithResidencyHint::Hot { role }(resident = hot by definition)PageKindfor downstream sub-pool partitioningNEUTRAL_FACTOR_STUB(0.5) for semantic / outcome_history / provenance_trust factors (dedicated integrations land in separate PRs)NEUTRAL_FACTOR_STUBpublic constant making the contract visibleDesign choices
working_set_snapshothelper rather than holding RwLock across the await. Same pattern as feat(genome): working-set-manager PR-5 — LocalWorkingSetManager auto-publishes via bus hook (+MessageBus borrow fix) #1362's bus_arc hook.Arc<dyn CandidateSource>per PR-3c's contract.Local{role=Bench/Cold/Frozen}for candidates outside the working set.What is deliberately deferred
Test plan
cargo test --lib --features metal,accelerate genome::recall_source_working_set— 7/7 pass:fetch_unregistered_persona_returns_empty_not_errorfetch_registered_empty_working_set_returns_emptyfetch_after_page_in_returns_resident_pages_as_hot_candidates— the payoff testtranslation_preserves_page_kind_for_sub_pool_partitioningtranslation_uses_neutral_factor_stubs_for_non_tier_factors— pins the contractsource_is_object_safe_for_arc_dyn_dispatchend_to_end_page_in_then_recall_returns_ranked_pool— full pipeline throughLocalDemandAlignedRecall::recallStack
LocalDemandAlignedRecallranking engineWorkingSetCandidateSource(closes the loop)🤖 Generated with Claude Code