Skip to content

feat(genome): demand-aligned-recall PR-3d — WorkingSetCandidateSource (genome stack payoff)#1378

Merged
joelteply merged 1 commit into
canaryfrom
feat/demand-aligned-recall-working-set-source-pr3d
May 17, 2026
Merged

feat(genome): demand-aligned-recall PR-3d — WorkingSetCandidateSource (genome stack payoff)#1378
joelteply merged 1 commit into
canaryfrom
feat/demand-aligned-recall-working-set-source-pr3d

Conversation

@joelteply
Copy link
Copy Markdown
Contributor

Summary

The architectural payoff of the genome stack. A persona's page_in calls populate the working set (my #1355); this source reads that same working set to surface "what's already hot" candidates that LocalDemandAlignedRecall (#1372 + #1374) ranks via the scoring function (#1371).

End-to-end loop now closed:

page_in(persona, page)
  → WorkingSet.pages updated
  → bus publishes PageFault (via #1362)
recall(query, ctx)
  → WorkingSetCandidateSource::fetch
    → working_set_snapshot
      → CandidateArtifact per resident page
        → LocalDemandAlignedRecall::rank
          → RankedPool

What lands

  • WorkingSetCandidateSource holding Arc<LocalWorkingSetManager>
  • CandidateSource::fetch impl that:
    • reads persona's working_set_snapshot
    • returns empty Vec on unregistered persona (legitimate cold-start signal, not error)
    • translates each ResidentPageCandidateArtifact with ResidencyHint::Hot { role } (resident = hot by definition)
    • preserves PageKind for downstream sub-pool partitioning
    • sets NEUTRAL_FACTOR_STUB (0.5) for semantic / outcome_history / provenance_trust factors (dedicated integrations land in separate PRs)
  • NEUTRAL_FACTOR_STUB public constant making the contract visible

Design choices

  • Snapshot via working_set_snapshot helper rather than holding RwLock across the await. Same pattern as feat(genome): working-set-manager PR-5 — LocalWorkingSetManager auto-publishes via bus hook (+MessageBus borrow fix) #1362's bus_arc hook.
  • Object-safe: works through Arc<dyn CandidateSource> per PR-3c's contract.
  • All resident pages map to Hot residency. A separate genome-catalog walker PR will add Local{role=Bench/Cold/Frozen} for candidates outside the working set.
  • Stub-0.5 factors documented inline + via constant. When embedding/sentinel/trust integrations land, they replace stubs without re-touching this file.

What is deliberately deferred

  • Genome catalog walker (Bench/Cold/Frozen tier sources)
  • Federation peer source
  • Embedding integration (semantic factor)
  • Sentinel outcome lookup (outcome_history factor)
  • Trust registry lookup (provenance_trust factor)

Test plan

  • cargo test --lib --features metal,accelerate genome::recall_source_working_set — 7/7 pass:
    • fetch_unregistered_persona_returns_empty_not_error
    • fetch_registered_empty_working_set_returns_empty
    • fetch_after_page_in_returns_resident_pages_as_hot_candidates — the payoff test
    • translation_preserves_page_kind_for_sub_pool_partitioning
    • translation_uses_neutral_factor_stubs_for_non_tier_factors — pins the contract
    • source_is_object_safe_for_arc_dyn_dispatch
    • end_to_end_page_in_then_recall_returns_ranked_pool — full pipeline through LocalDemandAlignedRecall::recall
  • No regressions across other 2822 lib tests
  • Pre-push gate clean

Stack

🤖 Generated with Claude Code

The architectural payoff of the genome stack lands here. A persona's
page_in calls populate the working set (#1355); this source reads
that same working set to surface "what's already hot" candidates
that LocalDemandAlignedRecall (#1372 + #1374) ranks via the scoring
function (#1371).

End-to-end loop closed:
  page_in(persona, page) → WorkingSet.pages updated → bus publishes
  PageFault (#1362) → recall(query, ctx) → working_set_snapshot →
  CandidateArtifact per resident page → rank() → RankedPool

What lands

- WorkingSetCandidateSource struct holding
  Arc<LocalWorkingSetManager>
- CandidateSource::fetch impl that:
  - reads persona's working_set_snapshot
  - returns empty Vec on unregistered persona (no error — cold-
    start signal callers may try federation)
  - translates each ResidentPage → CandidateArtifact with
    ResidencyHint::Hot { role } (resident = hot by definition)
  - preserves PageKind for downstream sub-pool partitioning
  - sets NEUTRAL_FACTOR_STUB (0.5) for semantic / outcome_history
    / provenance_trust factors (dedicated integrations land in
    separate PRs)
- NEUTRAL_FACTOR_STUB public constant for the contract

Design choices

- Snapshot the working set via the manager's working_set_snapshot
  helper (cloned) rather than holding the RwLock across the fetch
  await. Same pattern as #1362's bus_arc hook.
- Object-safe: works through Arc<dyn CandidateSource> per PR-3c's
  contract.
- All resident pages map to Hot residency. PR-3e (or a separate
  catalog walker PR) will add Local{role=Bench/Cold/Frozen} for
  candidates outside the working set but resident in the genome
  catalog.
- Stub-0.5 factors documented inline + via NEUTRAL_FACTOR_STUB
  constant. When the embedding / sentinel / trust integrations
  land, they replace the stubs without re-touching this file.

What is deliberately deferred

- Genome catalog walker (Bench/Cold/Frozen tier sources) — needs
  the catalog module
- Federation peer source — needs federation registry
- Embedding integration (semantic factor) — separate Lane H slice
- Sentinel outcome lookup (outcome_history factor) — sentinel PR
- Trust registry lookup (provenance_trust factor) — separate PR

Tests

7 new tests, all end-to-end with real LocalWorkingSetManager +
page_in calls:
- fetch_unregistered_persona_returns_empty_not_error
- fetch_registered_empty_working_set_returns_empty
- fetch_after_page_in_returns_resident_pages_as_hot_candidates —
  the payoff test
- translation_preserves_page_kind_for_sub_pool_partitioning —
  layer → layers, expert → experts, engram → engrams
- translation_uses_neutral_factor_stubs_for_non_tier_factors —
  pins the contract so embedding-integration PRs flip it
- source_is_object_safe_for_arc_dyn_dispatch — through PR-3c's
  Arc<dyn CandidateSource>
- end_to_end_page_in_then_recall_returns_ranked_pool — full
  pipeline: page_in → WorkingSetCandidateSource ::fetch →
  LocalDemandAlignedRecall::recall → RankedPool with the
  paged-in artifacts ranked correctly

7/7 pass. No regressions across other 2822 lib tests.

Stack

- #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager
- #1366 — DAR PR-1: pure types
- #1367 + #1370 — DAR PR-2: trait + composite types
- #1371 — DAR PR-3a: scoring function + per-factor curves
- #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine
- #1374 — DAR PR-3c: trait impl + CandidateSource seam
- THIS PR — DAR PR-3d: WorkingSetCandidateSource (the payoff)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@joelteply joelteply merged commit 90e0896 into canary May 17, 2026
3 checks passed
@joelteply joelteply deleted the feat/demand-aligned-recall-working-set-source-pr3d branch May 17, 2026 04:22
joelteply added a commit that referenced this pull request May 17, 2026
…1380)

Combines multiple CandidateSource impls into one, with optional
deduplication by artifact id. Sets up the extensibility seam so
future PRs (genome catalog walker, federation peer source,
must-include resolver) add sources without re-wiring
LocalDemandAlignedRecall.

What lands

- CompositeCandidateSource { sources, dedup }
- DedupPolicy::None — return all candidates from all sources (a
  single artifact may appear N times if N sources surface it).
  Useful for audit-trail callers.
- DedupPolicy::ByArtifactId — keep first occurrence per (kind,
  artifact_id) tuple in source-iteration order. Most callers want
  this (prevents double-counting a resident page that also
  surfaces via federation lookup).
- CandidateSource::fetch impl: fans out to all sources
  concurrently via futures::future::join_all, merges, dedups.
- new(sources, dedup) + with_default_dedup(sources) constructors.
- source_count() + dedup_policy() inspector methods.

Design choices

- futures::future::join_all for fan-out (concurrent, unbounded).
  Acceptable for ≤5 sources currently; federation peer counts may
  need bounding later — when that happens, this fn changes
  internals without breaking the trait.
- Dedup is configurable per composite. Most production wiring
  uses ByArtifactId; replay traces may use None for audit fidelity.
- Different PageKind with same artifact_id treated as distinct
  candidates (a layer-page reference and an engram-page reference
  happen to share the underlying artifact id; recall keeps them
  separate so the sub-pool partitioning is correct).
- Composite itself is object-safe — composites of composites
  valid for future hierarchical wiring.

What is deliberately deferred

- Source priority ordering — first-hit-wins per dedup. A future
  PR may add weighted merging.
- Per-source error isolation — fetch returns Vec, not Result. The
  underlying trait method also returns Vec; widening the trait
  would be a separate concern.
- Bounded concurrent fan-out — join_all is unbounded. Fine for
  the current source count; needs revisit when federation peers
  scale.

Tests

9 new tests pin the composite's behaviors:
- empty_composite_returns_empty_vec — no-error empty contract
- single_source_composite_passes_through — degenerate case
- fan_out_invokes_every_source_exactly_once — per-call accounting
- merge_preserves_source_iteration_order — dedup correctness
  depends on this
- dedup_none_preserves_all_duplicates
- dedup_by_artifact_id_keeps_first_occurrence_only
- dedup_treats_different_page_kinds_as_distinct
- with_default_dedup_uses_by_artifact_id
- composite_is_object_safe_as_dyn_candidate_source

9/9 pass. No regressions across other 2834 lib tests.

Stack

- #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager
- #1366 — DAR PR-1: pure types
- #1367 + #1370 — DAR PR-2: trait + composite types
- #1371 — DAR PR-3a: scoring function + per-factor curves
- #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine
- #1374 — DAR PR-3c: trait impl + CandidateSource seam
- #1378 — DAR PR-3d: WorkingSetCandidateSource
- THIS PR — DAR PR-3e: CompositeCandidateSource (extensibility seam)
- NEXT — DAR PR-3f or later: catalog walker + federation source +
  must-include resolver, all composing through this PR's seam

Co-authored-by: Test <test@test.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
joelteply added a commit that referenced this pull request May 17, 2026
#1382)

Resolves CapabilityQuery.must_include hard pins as candidates per
GENOME-FOUNDRY-SENTINEL Part 7: "Hard pins — recall MUST include
these in the RankedPool even if their score is low. Used for
persona-private LoRA layers and sticky engrams."

Plays through the composite seam shipped in PR-3e: wired AFTER a
resident source like WorkingSetCandidateSource with ByArtifactId
dedup, must-include items that ARE resident get the resident
source's Hot residency + factor data; must-include items NOT
resident get this source's NotResident placeholder (still ranked,
just lower combined score).

What lands

- MustIncludeCandidateSource — zero-state unit struct (no Arc state
  needed; the source is pure-function over the query)
- CandidateSource::fetch impl that:
  - reads query.must_include Vec<ArtifactRef>
  - maps each variant (LoRALayer / MoEExpert / Engram) to a
    CandidateArtifact with the appropriate PageKind
  - marks every must-include candidate as ResidencyHint::
    NotResident { acquirable_from: SentinelRefinement }
  - uses NEUTRAL_FACTOR_STUB (0.5) for the three non-tier factors,
    same convention as WorkingSetCandidateSource (PR-3d)

Recommended composite wiring

  let composite = CompositeCandidateSource::with_default_dedup(vec![
      Arc::new(WorkingSetCandidateSource::new(mgr)),     // Hot first
      Arc::new(MustIncludeCandidateSource::new()),       // Pins
      // future: catalog walker, federation source
  ]);

Spec contract met: every hard-pinned artifact surfaces in the
RankedPool; if it's resident, it gets full residency-aware score;
if not, it still appears (at lower combined) so composition can
see "this was pinned but isn't here yet — schedule the foundry."

Tests

6 new tests:
- empty_must_include_returns_empty_candidates (no-error empty
  contract)
- variant_mapping_preserves_page_kind (LoRALayer/MoEExpert/Engram
  variants → PageKind mapping)
- must_include_marks_candidates_as_not_resident
- factors_use_neutral_stubs_consistent_with_working_set_source
- source_is_object_safe_for_dyn_dispatch
- composite_with_dedup_resident_wins_must_include_for_pinned_hot_
  artifact — the architectural payoff: resident pin keeps Hot,
  non-resident pin gets NotResident, both appear in merged Vec

6/6 pass. No regressions across other 2873 lib tests.

Stack

- #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager
- #1366 — DAR PR-1: pure types
- #1367 + #1370 — DAR PR-2: trait + composite types
- #1371 — DAR PR-3a: scoring function + per-factor curves
- #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine
- #1374 — DAR PR-3c: trait impl + CandidateSource seam
- #1378 — DAR PR-3d: WorkingSetCandidateSource (working-set source)
- #1380 — DAR PR-3e: CompositeCandidateSource (extensibility seam)
- THIS PR — DAR PR-3f: MustIncludeCandidateSource (hard-pin source)

Co-authored-by: Test <test@test.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant