feat(governor): select policy by hardware fingerprint#1352
Conversation
|
Closing as duplicate of #1351 for the same governor PR-3a lane. I am carrying the stricter review points over there instead: empty applies_to should be an explicit config error, and multiple matches should be explicit ambiguity unless the policy format grows a real priority field. |
joelteply
left a comment
There was a problem hiding this comment.
Reviewed #1352. Solid design — the AmbiguousPolicy refusal is strictly better than my (now-closed) #1351's longest-applies_to tiebreaker, and the hardware_fingerprint() public helper is a useful diagnostics surface I didn't build. Approving direction.
Two follow-up notes (NOT blocking — these are PR-3b/PR-3c territory):
-
EmptyAppliesTo per-policy — a policy file with empty applies_to causes select_policy to ALWAYS return Err (because policy_matches_hardware returns Err for that policy, which propagates through filter_map). That breaks any first-boot bootstrap case where a fallback 'no-applies_to means wildcard' might land. Worth surfacing — design call: should empty applies_to be (a) treated as wildcard (my closed PR's behavior), (b) treated as a config error per current implementation, or (c) treated as no-op (skip the policy without erroring the selection)? Current behavior is strictest; (c) might be most permissive without losing safety.
-
MalformedRange loses parse-error reason — when range bounds fail to parse, the error is just MalformedRange { token } without the underlying ParseIntError reason. Operator sees 'malformed' but not 'parse error: invalid digit'. Cheap improvement: add reason: String field (could land in PR-3c or a follow-up). My closed PR had this; happy to PR the addition if useful.
Going to rebase my local PR-3b LocalSubstrateGovernor (arc_swap + initial_policy + on_hardware_detected impl + 16 tests including concurrent-read-during-write) onto your selector API as soon as this merges. The arc_swap dep + ~400 LOC + tests are ready locally.
Approving the merge direction in airc; no formal GH review since I'm marked as the same-org committer and the queue is moving fast — codex's babysitting is closing PRs faster than formal reviews would.
|
Reopening this as the PR-3a merge target. #1351 was also closed, and this branch is the stricter API the follow-on PR-3b work said it would rebase onto: stable hardware_fingerprint diagnostics, EmptyAppliesTo, NoMatchingPolicy, and AmbiguousPolicy. No implicit wildcard and no implicit priority. |
|
Proof after reopen:
This PR keeps the no-silent-fallback contract: empty applies_to is EmptyAppliesTo, unknown constraints are typed errors, malformed ranges are typed errors, zero matches are NoMatchingPolicy with fingerprint, and >1 match is AmbiguousPolicy. The selector does not invent a default policy or implicit priority. |
Stacks on #1352 (codex's PR-3a policy_selector, MERGED). Per GENOME-FOUNDRY-SENTINEL #1327 Part 11. LocalSubstrateGovernor is the reference impl of the SubstrateGovernor trait (from #1345 PR-1). Holds the live policy behind arc_swap for wait-free reads; mutex-protected snapshot history for telemetry. What ships in src/workers/continuum-core/src/governor/local.rs: - LocalSubstrateGovernor struct: Arc<ArcSwap<GovernorPolicy>> for policy + Mutex<SnapshotState> for cascade-transition-count + recent-signals ring - new(initial_policy) constructor — ready to serve current_policy() immediately - set_candidates(Vec<PolicyFile>) — file watcher (PR-3d) will call this on fs change events; for PR-3b, set manually - try_hardware_detected(hw) → Result<(), PolicySelectionError> — fallible variant for callers that want the typed error - on_hardware_detected(hw) — trait method, swallows errors per spec (logs/telemetry surface them separately) - on_pressure_signal(signal) — records into ring (PR-3c adds threshold + cascade logic; PR-3b only records) - snapshot() → GovernorSnapshot — telemetry consumer reads this - candidate_count() — diagnostic for 'did the file watcher load anything?' Concurrency model (matches spec's 'never blocks reads'): - Reads: arc_swap.load_full() → Arc<GovernorPolicy> clone (wait-free) - Writes: arc_swap.store(Arc::new(new_policy)) + mutex on snapshot state for transition-count bump (~µs hold) - Tests prove the wait-free guarantee: many_concurrent_reads_dont_block + concurrent_read_during_write_sees_consistent_snapshot What this PR DOES NOT do: - Cascade state machine + threshold/hysteresis (PR-3c) - File watcher / hot reload (PR-3d) - PressureBroker subscription wiring (PR-4) - Built-in default policy fallback (caller handles NoMatchingPolicy) Failure-mode discipline: - on_hardware_detected with no matching candidate KEEPS previous policy (trait swallows error per spec — operator monitors via snapshot.cascade_transition_count which stays unchanged on Err) - on_hardware_detected with empty candidates is a no-op (first-boot before file watcher loads anything — governor still serves initial_policy) - cascade_transition_count increments per PUBLISH, not per call — failed selections don't count - on_pressure_signal does NOT bump cascade_transition_count in PR-3b (test pins this so PR-3c lands the threshold logic together) Tests: 16 passing on cargo test --lib --features metal,accelerate governor::local:: (79 total governor:: across PR-1/PR-2/PR-3a/PR-3b) - new() serves initial policy immediately - candidate_count reflects set_candidates - on_hardware_detected publishes matching policy - try_hardware_detected returns NoMatchingPolicy err - on_hardware_detected no-match KEEPS previous policy - on_hardware_detected empty candidates no-op - Successive hardware_detected publishes multiple times - on_pressure_signal records signal - recent_signals ring capped at RECENT_SIGNALS_CAPACITY=32 (FIFO eviction) - snapshot includes policy + signals - cascade_transition_count increments per publish - cascade_transition_count UNCHANGED on no-match - on_pressure_signal does NOT transition in PR-3b (PR-3c adds it) - many_concurrent_reads_dont_block (Arc<Self> + 16 threads × 1000 reads each) - concurrent_read_during_write_sees_consistent_snapshot (writer mutates + reader observes Arc snapshots that are always one of {1, 2, 8} — no torn read) - current_policy returns same Arc when no writes (Arc::ptr_eq) Added deps: arc-swap = '1.7' (tiny crate, no transitive deps). Coordination: ceded my own PR-3a (#1351 closed) in favor of codex's #1352 which has stricter AmbiguousPolicy refusal + hardware_fingerprint diagnostic surface. This PR-3b rebased onto codex's policy_selector API (arg order: select_policy(policies, hw), not (hw, policies)) + imports updated. Stack: - #1335 hw_probe (MERGED) - #1345 PR-1 governor-types (MERGED) - #1350 PR-2 TOML loader (MERGED) - #1352 PR-3a policy_selector (codex's, MERGED) - This PR (PR-3b): LocalSubstrateGovernor + arc_swap publish - Future PR-3c: cascade state machine + hysteresis (5 steps; restore- speculation-one-step-later anti-oscillation rule per spec) - Future PR-3d: file watcher (notify crate) - Future PR-4: PressureBroker → governor wiring VDD evidence N/A — pure-state impl. Evidence with PR-3c when the cascade is wired + with PR-4 when actual pressure signals flow. Co-authored-by: Test <test@test.com>
Stacks on canary post-#1360 merge. PR-3c2 wired cascade evaluator into on_pressure_signal to update cascade_step. This PR-3c3 ships apply_cascade_step_to_policy — the pure function that ACTUALLY transforms tier_sizes/cadence/concurrency/ speculation/federation/consolidation per the cascade step. Per spec §'Adjustment Cascade' table: - Step 0: unchanged (normal operation) - Step 1: speculation_aggressiveness drops one notch toward Off (Aggressive → Balanced → Conservative → Off → Off) - Step 2: cumulative + personas_concurrent -= 1 (floor 1) + defer non-realtime (cadence_multipliers.delayed/.background = max(current, 2.0)) - Step 3: cumulative + tier_sizes.l1_lora_layers + l1_kv_tokens shrunk to 75% (floor 1) - Step 4: cumulative + federation_pull_cadence.pull_cadence_seconds = MAX_FEDERATION_PULL_CADENCE_SECONDS (3600s = once-per-hour) - Step 5: cumulative + consolidation_schedule = Manual (operator must explicitly trigger; substrate stops on its own under max pressure) Transformations are CUMULATIVE — step N includes all transformations from steps 1..N. Caller passes BASE policy (cascade_step=0) and step; function returns a NEW policy with cascade_step + transformations applied. Caller is responsible for bumping policy_version + updating committed_at_ms at publish time. Pure function — no I/O, no state, no globals. Deterministic. Anti-oscillation note (caller responsibility, documented in fn docstring): the spec's 'restore-speculation-one-step-later' rule lives in the WIRING layer (LocalSubstrateGovernor follow-up), not this pure transformation. When retreating N → N-1, caller applies step N-1 for everything EXCEPT speculation, which uses step N for one more cycle. This separation keeps apply_cascade_step_to_policy a clean deterministic mapping. Also documented (test pins this): apply_cascade_step_to_policy is NOT reversible from a transformed policy. apply(transformed, 0) does NOT restore base — the caller must hold the original base separately and re-apply step 0 from it. LocalSubstrateGovernor will need to evolve to store base + active separately (PR-3c4). Constants: - MAX_FEDERATION_PULL_CADENCE_SECONDS = 3600 (once-per-hour ceiling) Pinned by test to catch silent tuning. Tests: 46 passing on cargo test --lib --features metal,accelerate governor::cascade:: (30 from PR-3c1 + 16 new) NEW (16) for apply_cascade_step_to_policy: - step 0 == base except cascade_step (identity) - step 1 drops Aggressive → Balanced - step 1 covers full speculation ladder (4 variants) - step 2 personas-1 + cumulative speculation drop - step 2 personas floor at 1 (defensive) - step 2 stretches non-realtime cadence (delayed + background → 2.0) - step 2 doesn't shrink already-stretched cadence (max-not-set semantics) - step 3 shrinks l1 by 25% (8→6, 16384→12288) - step 3 l1 floors at 1 (1*0.75=0.75→0→max(0,1)=1) - step 4 federation_pull_cadence_seconds = MAX (60→3600) - step 5 consolidation = Manual - step 5 cumulative — all prior transformations applied - step > MAX clamps to MAX (defensive against caller bugs) - determinism - not reversible from transformed (documented limitation, test pinned) - MAX_FEDERATION_PULL_CADENCE_SECONDS const pinned Stack: - #1345 PR-1 governor-types (MERGED) - #1350 PR-2 TOML loader (MERGED) - #1352 PR-3a policy_selector (MERGED) - #1354 PR-3b LocalSubstrateGovernor (MERGED) - #1356 PR-3c1 cascade evaluator (MERGED) - #1360 PR-3c2 cascade wiring + time-in-step gate (MERGED) - This PR (PR-3c3): apply_cascade_step_to_policy field rewrites - Future PR-3c4: wire apply_cascade_step_to_policy into LocalSubstrateGovernor + restore-speculation-one-step-later semantics + base-vs-active policy split - Future PR-3d: file watcher (notify crate) - Future PR-4: PressureBroker → governor wiring VDD evidence N/A — pure transformation. Evidence with PR-3c4 wiring + PR-4 + downstream consumers reading the throttled policy. Coordination: explicit claim posted to airc 00:25Z; codex on orthogonal VDD work per their 00:25:13Z broadcast. No collision. Co-authored-by: Test <test@test.com>
…ase/active split + restore-speculation-one-step-later Stacks on #1364 (PR-3c3 apply_cascade_step_to_policy, MERGED). PR-3c3 shipped the pure function. PR-3c4 wires it into LocalSubstrateGovernor with the base-vs-active policy split + the spec's restore-speculation-one-step-later anti-oscillation rule. What changed in local.rs: - LocalSubstrateGovernor.base_policy: Mutex<GovernorPolicy> field added. Holds the canonical un-throttled policy (cascade_step always 0). Cascade transitions re-derive active from base via apply_cascade_step_to_policy, never from the already-throttled current. This addresses PR-3c3's not-reversible-from-transformed documented limitation. - SnapshotState.pending_speculation_retreat: bool added. Tracks whether the cascade just retreated; if true, the NEXT Hold or Retreat restores speculation to the lower-step value. The first retreat keeps speculation at the higher-step (pre-retreat) value for one more cycle. - new() initializes base_policy from the supplied initial_policy (cascade_step normalized to 0 on the base; active keeps the supplied cascade_step). - try_hardware_detected() refreshes base_policy + resets cascade (step 0, last_step_change_ms now, pending_speculation_retreat cleared). New hardware = fresh start; existing pressure state discarded. - on_pressure_signal() rewired: * Same time-in-step gate as PR-3c2 (Advance from step > 0 within MIN_TIME_IN_STEP_MS Hold; emergency bypasses; retreat never gated) * On step change: clone base_policy + call apply_cascade_step_to_policy + bump policy_version + update committed_at_ms * On retreat: also apply prev_step's speculation to next_policy (one-step-later semantics) + set pending_speculation_retreat * On Advance after pending-retreat: clear marker (new pressure re-throttles speculation immediately) * On Hold with pending marker: deliver the restoration (publish new policy with current_step's speculation; clear marker) Restore-speculation-one-step-later rationale (from spec): Speculation thrash is the most user-visible cascade flapping. By keeping speculation throttled for ONE EXTRA cycle after the cascade retreats, we dampen the most observable form of oscillation while letting the rest of the policy (tier sizes, cadence, concurrency) restore immediately. The cost is one cycle of slightly-throttled speculation; the benefit is no observable flicker between Aggressive and Balanced (or whatever pair the cascade is bouncing between). Failure-mode discipline: - Base policy is the ONLY source of truth for transformations. Active is always derived; never mutated in place. - Restore-one-step-later is typed (bool marker, not a magic time comparison or a sentinel value). - Hardware change wipes pending retreat marker — new hardware = clean slate; old cascade state doesn't bleed into new policy. Tests: 29 passing on cargo test --lib --features metal,accelerate governor::local:: (22 prior + 7 new for PR-3c4) NEW (7): - advance_derives_active_from_base_with_step_transformations - emergency_advance_applies_full_throttle_transformations (full step-5 cumulative: tier_sizes shrunk, federation maxed, consolidation Manual, speculation dropped, personas-1) - retreat_holds_speculation_for_one_more_cycle (anti-oscillation rule pinned: Advance 0→1 drops Aggr→Balanced; Retreat 1→0 KEEPS Balanced; next Hold RESTORES Aggressive) - advance_during_pending_retreat_clears_marker - hardware_detected_refreshes_base_and_resets_cascade - advance_then_retreat_returns_to_base_values_modulo_speculation_dampening (proves derive-from-base prevents compounding transformations — was PR-3c3's not-reversible warning) - (helpers: policy_with_l1, policy_with_l1_nvidia) Stack: - #1345 / #1350 / #1352 / #1354 / #1356 / #1360 / #1364 — Lane H PRs MERGED - This PR (PR-3c4): wire apply_cascade_step_to_policy + base/active split + restore-speculation-one-step-later - Future PR-3d: file watcher (notify crate) — hot-reload policy file changes via set_candidates - Future PR-4: PressureBroker → governor wiring (subscribe to typed pressure events from broker) VDD evidence N/A — wiring + state machine. Evidence with PR-4 + harness measurements when real pressure flows + downstream consumers read throttled policy fields. Coordination: explicit claim posted 00:40Z; codex on demand-aligned- recall PR-1 per their 00:40:22Z broadcast. claude-tab-1 on whatever- next. No collision.
… bridge
Pure-function bridge between PressureBroker's PressureAlert surface
(disk/memory pool eviction events) and the governor's typed
PressureSignal cascade input. Per GENOME-FOUNDRY-SENTINEL.md Part 11
line 1121: "PressureBroker informs the SubstrateGovernor. Pressure
signals from the broker drive the governor's adjustment cascade."
Scope:
- `alert_to_signal(&PressureAlert) -> Option<PressureSignal>` — pure
mapping. High/Critical tier → SystemMemHigh{used_pct}; Normal/
Warning/unknown → None.
- `governor_alert_sink(Arc<dyn SubstrateGovernor>) -> AlertSink` —
factory that wraps a governor as an AlertSink the broker can register
via `PressureBroker::add_alert_sink`. Sink derives the signal and
forwards via `governor.on_pressure_signal` when Some; drops when None.
NOT in this PR (deferred to PR-5):
- Wiring the sink into PressureBrokerModule's boot path. The bridge is
the data-side primitive; the wiring is a separate concern.
- Pool-name-aware mapping (vram → VRAMHigh, etc.). Today's broker pools
are all memory-adjacent (Docker disk, HF cache, future VRAM via
GpuMemoryManager); SystemMemHigh is the conservative single-mapping
the cascade reacts to identically. Refinement when pool tier_name
conventions stabilize.
Discipline:
- No silent default-on-error. Mapping is total — every alert maps to
either Some(signal) or None explicitly.
- Pressure clamped to [0.0, 1.0] before percent conversion so transient
over-budget snapshots map to 100% and negative artifacts map to 0%
rather than wrapping via `as u8`.
- Sink forwards via `Arc<dyn SubstrateGovernor>` (object-safe trait) so
the bridge does not depend on LocalSubstrateGovernor concretely.
Tests (14, all passing):
- normal/warning/unknown tiers -> None (4 tests)
- high/critical tiers -> SystemMemHigh with rounded used_pct (3 tests)
- pressure clamping above 1.0 + below 0.0 + rounding (3 tests)
- sink forwarding high/critical + non-forwarding normal/warning (4 tests)
- sink survives construction-scope drop + multi-call ordering (2 tests)
Lane H 8-PR stack progress: PR-1 (#1330/1331) -> PR-2 (#1345) -> PR-3a
(#1352) -> PR-3b (#1354) -> PR-3c1 (#1356) -> PR-3c2 (#1360) -> PR-3c3
(#1364) -> PR-3c4 (#1365) -> **PR-4 (this PR)**. PR-3d governor file
watcher in flight from codex on parallel branch (no overlap).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… bridge
Pure-function bridge between PressureBroker's PressureAlert surface
(disk/memory pool eviction events) and the governor's typed
PressureSignal cascade input. Per GENOME-FOUNDRY-SENTINEL.md Part 11
line 1121: "PressureBroker informs the SubstrateGovernor. Pressure
signals from the broker drive the governor's adjustment cascade."
Scope:
- `alert_to_signal(&PressureAlert) -> Option<PressureSignal>` — pure
mapping. High/Critical tier → SystemMemHigh{used_pct}; Normal/
Warning/unknown → None.
- `governor_alert_sink(Arc<dyn SubstrateGovernor>) -> AlertSink` —
factory that wraps a governor as an AlertSink the broker can register
via `PressureBroker::add_alert_sink`. Sink derives the signal and
forwards via `governor.on_pressure_signal` when Some; drops when None.
NOT in this PR (deferred to PR-5):
- Wiring the sink into PressureBrokerModule's boot path. The bridge is
the data-side primitive; the wiring is a separate concern.
- Pool-name-aware mapping (vram → VRAMHigh, etc.). Today's broker pools
are all memory-adjacent (Docker disk, HF cache, future VRAM via
GpuMemoryManager); SystemMemHigh is the conservative single-mapping
the cascade reacts to identically. Refinement when pool tier_name
conventions stabilize.
Discipline:
- No silent default-on-error. Mapping is total — every alert maps to
either Some(signal) or None explicitly.
- Pressure clamped to [0.0, 1.0] before percent conversion so transient
over-budget snapshots map to 100% and negative artifacts map to 0%
rather than wrapping via `as u8`.
- Sink forwards via `Arc<dyn SubstrateGovernor>` (object-safe trait) so
the bridge does not depend on LocalSubstrateGovernor concretely.
Tests (14, all passing):
- normal/warning/unknown tiers -> None (4 tests)
- high/critical tiers -> SystemMemHigh with rounded used_pct (3 tests)
- pressure clamping above 1.0 + below 0.0 + rounding (3 tests)
- sink forwarding high/critical + non-forwarding normal/warning (4 tests)
- sink survives construction-scope drop + multi-call ordering (2 tests)
Lane H 8-PR stack progress: PR-1 (#1330/1331) -> PR-2 (#1345) -> PR-3a
(#1352) -> PR-3b (#1354) -> PR-3c1 (#1356) -> PR-3c2 (#1360) -> PR-3c3
(#1364) -> PR-3c4 (#1365) -> **PR-4 (this PR)**. PR-3d governor file
watcher in flight from codex on parallel branch (no overlap).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
) Pure-function bridge between PressureBroker's PressureAlert surface (disk/memory pool eviction events) and the governor's typed PressureSignal cascade input. Per GENOME-FOUNDRY-SENTINEL.md Part 11 line 1121: "PressureBroker informs the SubstrateGovernor. Pressure signals from the broker drive the governor's adjustment cascade." Scope: - `alert_to_signal(&PressureAlert) -> Option<PressureSignal>` — pure mapping. High/Critical tier → SystemMemHigh{used_pct}; Normal/ Warning/unknown → None. - `governor_alert_sink(Arc<dyn SubstrateGovernor>) -> AlertSink` — factory that wraps a governor as an AlertSink the broker can register via `PressureBroker::add_alert_sink`. Sink derives the signal and forwards via `governor.on_pressure_signal` when Some; drops when None. NOT in this PR (deferred to PR-5): - Wiring the sink into PressureBrokerModule's boot path. The bridge is the data-side primitive; the wiring is a separate concern. - Pool-name-aware mapping (vram → VRAMHigh, etc.). Today's broker pools are all memory-adjacent (Docker disk, HF cache, future VRAM via GpuMemoryManager); SystemMemHigh is the conservative single-mapping the cascade reacts to identically. Refinement when pool tier_name conventions stabilize. Discipline: - No silent default-on-error. Mapping is total — every alert maps to either Some(signal) or None explicitly. - Pressure clamped to [0.0, 1.0] before percent conversion so transient over-budget snapshots map to 100% and negative artifacts map to 0% rather than wrapping via `as u8`. - Sink forwards via `Arc<dyn SubstrateGovernor>` (object-safe trait) so the bridge does not depend on LocalSubstrateGovernor concretely. Tests (14, all passing): - normal/warning/unknown tiers -> None (4 tests) - high/critical tiers -> SystemMemHigh with rounded used_pct (3 tests) - pressure clamping above 1.0 + below 0.0 + rounding (3 tests) - sink forwarding high/critical + non-forwarding normal/warning (4 tests) - sink survives construction-scope drop + multi-call ordering (2 tests) Lane H 8-PR stack progress: PR-1 (#1330/1331) -> PR-2 (#1345) -> PR-3a (#1352) -> PR-3b (#1354) -> PR-3c1 (#1356) -> PR-3c2 (#1360) -> PR-3c3 (#1364) -> PR-3c4 (#1365) -> **PR-4 (this PR)**. PR-3d governor file watcher in flight from codex on parallel branch (no overlap). Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Failure discipline
Scope
Test plan