Skip to content

fix(reconciler): debounce pool scale_check before spawning new sessions - #6

Open
austinborn wants to merge 2 commits into
mainfrom
fix-pool-scale-check-debounce
Open

fix(reconciler): debounce pool scale_check before spawning new sessions#6
austinborn wants to merge 2 commits into
mainfrom
fix-pool-scale-check-debounce

Conversation

@austinborn

Copy link
Copy Markdown

Summary

  • Adds a process-scoped per-template scale_check confirm window to the pool reconciler. Reconciler call sites now read the minimum count over the last N=3 samples instead of acting on the raw value, so a single-tick flap to a positive value can no longer drive a fresh pool spawn that has nothing to claim.
  • The cold-start window (history smaller than N) passes the raw count through, so legitimate new demand still spawns on the first observation. Steady-state demand (every sample ≥ 1) is unchanged once the window is full — min over [1,1,1] is 1.
  • Templates whose probe failed this tick (poolScaleCheckPartialTemplates) are pass-through: a probe failure (raw count = 0) is not a real observation of demand and must not poison the window. retainScaleCheckPartialPoolDesired continues to own session preservation across partial-failure ticks.

Motivation

The pathology this addresses: an operator observed a pool slot in a ~66s respawn loop. scale_check_count flipped 0 ↔ 1 in the reconciler trace, but external probes of the exact same bd ready --metadata-field gc.routed_to=<pool> --unassigned --exclude-type=epic --json --limit 0 | jq length always returned 0. Each transient 1 raised new-tier demand, a session bead was created, the session ran the work_query, found no claimable bead, and drained. ~170 ghost spawns in 4h on a single slot.

Whatever upstream cause briefly produced an unassigned routed bead inside one tick (most plausibly a release-and-restore round trip on the assignee within a single reconcile), the spawn loop is the symptom this PR collapses. The orphan-release path inside releaseOrphanedPoolAssignmentsWhenSnapshotsComplete already runs liveOpenSessionAssignmentExists against the live session label set (the protection introduced in PR gastownhall#1856 / commit e796e7d9), so pinning a release containing that fix wouldn't have changed behavior here — the loop has to be filtered downstream of demand observation.

What changed

  • cmd/gc/pool_scale_check_debouncer.go (new) — singleton holding per-template ring buffer + the debouncePoolScaleCheckCounts entry point. Mutex-protected; tests can opt out (window=1) or opt in (window=3+) via setPoolScaleCheckDebouncerWindowForTesting.
  • cmd/gc/pool_desired_state.go — adds ComputePoolDesiredStatesDebounced and ComputePoolDesiredStatesDebouncedTraced wrappers. The pure ComputePoolDesiredStates{,Traced} entry points are unchanged so existing tests of the compute kernel remain valid.
  • cmd/gc/build_desired_state.go (1 site), cmd/gc/city_runtime.go (3 sites — beadReconcileTick fallback, controlDispatcherTick, loadDemandSnapshot), cmd/gc/cmd_start.go (1 site) — swap to the Debounced variants. These are every reconciler-driven path that consumes result.ScaleCheckCounts.
  • cmd/gc/main_test.gosetPoolScaleCheckDebouncerWindowForTesting(1) in TestMain so existing single-call spawn assertions don't have to seed history. Debouncer-specific tests opt back in via withPoolScaleCheckDebouncerWindow(t, 3).
  • cmd/gc/pool_scale_check_debouncer_test.go (new) — covers: cold-start pass-through, sustained demand, single-tick flap suppression (the canonical pathology), partial-template pass-through (and that partials don't poison the window), per-template isolation, window=1 is a no-op, and reset semantics.

Trade-offs

  • Cold-start latency: zero — the first N-1 samples pass through.
  • Steady-state new-demand latency: bounded by N × tick_interval. At the typical ~10–30s tick cadence, that's 30–90s before sustained new demand confirms. In-flight new-tier sessions from prior ticks continue to count as spent demand via poolInFlightNewRequests during this wait, so the slot doesn't sit idle.
  • Window is hard-coded at N=3. Operators who want to tune it can layer a [daemon] config knob in a follow-up PR.

Test plan

  • Targeted: go test ./cmd/gc/ -run 'TestPoolScaleCheckDebouncer' -count=1 — six new tests, all pass.
  • Pool / reconciler regression sweep: go test ./cmd/gc/ -run 'TestComputePoolDesiredStates|TestComputeAwakeSet|TestRetainScaleCheckPartial|TestBuildDesiredState|TestSessionReconciler|TestReleaseOrphaned' -count=1 — all pass.
  • Full package: go test ./cmd/gc/... -count=1 -timeout 600s. Two failures (TestRunStartDriftCheck_RestartReturnsContinue, TestDoStartJSONAlreadyRunningSupervisorKeepsStdoutJSONOnly/drift_restart) reproduce identically on a clean origin/main (8028e93) — both expect /proc/<pid>/exe to be readable, which doesn't exist on macOS. Pre-existing macOS-specific, unrelated to this PR.
  • Field validation: install the binary, restart the supervisor that's hosting the looping slot, watch the slot 2 spawn cadence go from ~66s to no recurring respawns within the first 1–2 min of steady-state.

Generated by the operator's software factory.
• City: `factory-main` · Agent: `local-core.builder-2`
• On behalf of: @austinborn

Adds a process-scoped per-template scale_check confirm window to the
pool reconciler. Each tick records the raw count for every pool
template; new-demand consumers (ComputePoolDesiredStatesDebounced{,Traced})
see the minimum count across the last N=3 samples once the window is
full, and the raw count while it's still warming up. This collapses
the single-tick scale_check flap class of pathology while preserving
cold-start responsiveness and steady-state demand semantics.

Behavior:
  * Templates whose probe failed this tick (poolScaleCheckPartialTemplates)
    pass through the debouncer unchanged. Recording a probe failure
    (count=0) into the window would force the next two ticks to 0 even
    after a steady-state of 1, so the existing
    retainScaleCheckPartialPoolDesired path is left to handle the
    partial-failure session-preservation case.
  * Window=3 is hard-coded to keep this PR small. A daemon config knob
    can be layered on later if operators want to tune it.
  * Tests pin window=1 in TestMain so existing single-call assertions
    about spawn behavior continue to hold. Debouncer tests opt back in
    via setPoolScaleCheckDebouncerWindowForTesting.

Motivation:
  An operator observed builder pool slot 2 in a ~66s respawn loop —
  scale_check_count flipped 0→1→0 in the reconciler trace even though
  external `bd ready --metadata-field gc.routed_to=<pool> --unassigned`
  probes always returned 0. Each transient 1 triggered an anonymous
  new-tier spawn; the spawned session ran the work_query, found no
  claimable bead, and drained. Debouncing the count across a small
  window eliminates the cycle without changing the spawn-on-real-demand
  contract.

Touch points:
  * cmd/gc/pool_scale_check_debouncer.go (new) — singleton + window logic.
  * cmd/gc/pool_desired_state.go — adds ComputePoolDesiredStatesDebounced
    and ComputePoolDesiredStatesDebouncedTraced wrappers; pure entry
    points are unchanged.
  * cmd/gc/build_desired_state.go, city_runtime.go, cmd_start.go — swap
    the reconciler call sites to the Debounced variants.
  * cmd/gc/main_test.go — pin window=1 for tests.
  * cmd/gc/pool_scale_check_debouncer_test.go (new) — cold-start,
    sustained, single-tick flap, partial pass-through, per-template
    isolation, and reset tests.

Generated by the operator's software factory.
City: factory-main · Agent: local-core.builder-2
On behalf of: @austinborn
Co-Authored-By: factory-bot <factory-bot@austinborn.invalid>
…sfied

`h := append(d.history[template], count)` triggers gocritic's appendAssign
because the resulting slice header is bound to a different variable than the
input. The intent is identical when written as two statements — fetch the
existing history into h, then append onto h — but the linter is happy.

No behavior change.

Generated by the operator's software factory.
City: factory-main · Agent: local-core.builder-2
On behalf of: @austinborn
Co-Authored-By: factory-bot <factory-bot@austinborn.invalid>
austinborn pushed a commit that referenced this pull request Jul 24, 2026
gastownhall#4151)

## What

Makes the reconciler's pool orphan-release conditional on the claim it
observed: prefer the store's atomic `ReleaseIfCurrent` CAS
(status/assignee) and, where a store cannot conditionally release,
re-verify the snapshot with a live read immediately before the write
plus a verify-after read that logs a raced claim loudly — replacing the
previous unconditional `Update(assignee:"", status:"open")`.

## Why

The 2026-07-11 sqlite infra-store concurrency audit flagged this as
defect #6 (MED): the controller's orphan path
(`releaseOrphanedPoolAssignment`, `cmd/gc/pool_session_name.go`) was a
check-then-act TOCTOU — snapshot assigned work, run staleness gates
(plus a potentially slow detached probe), then write an unconditional
release. A legitimate re-claim landing in that window (crash recovery
racing a live or revived claimer) was silently clobbered back to
open/unassigned, with exit 0 everywhere and no runtime signal.
Compounding it, conditional release was dead on the sqlite infra
backend: `bd sql` is unsupported there (empirically confirmed by the
audit), so `BdStore.ReleaseIfCurrent` fail-loud dies — and the orphan
path never attempted it anyway.

Audit context: the same hammer ran ~5,900 real cross-process bd
operations with 640/640 claim races producing exactly one winner — the
engine-level claim CAS is sound. This gc-side unconditional write was
the layer above it that could undo a legitimately won claim. A red-team
harness against a real bd v1.1.0 sqlite workspace reproduced the clobber
deterministically on unpatched code (numbers below).
`internal/beads/bdstore.go` also gains a SEAM comment marking exactly
where bd's native conditional-release verb slots in once it ships
(callers already treat `ErrConditionalReleaseUnsupported` as "take the
recheck fallback", so no caller changes will be needed).

## Verification

TDD unit tests (5 new tests covering: CAS preferred when supported, lost
CAS race means no unconditional retry, unsupported store rechecks before
write, raced claim after the fallback write is logged loudly,
uncontended orphan still releases):

```
$ go test ./cmd/gc/ -run 'TestReleaseOrphanedPoolAssignments|TestWorkAssignmentReleaseWorkBead' -count=1
ok      github.com/gastownhall/gascity/cmd/gc   1.709s
$ go vet ./cmd/gc/ ./internal/beads/    # clean
```

Red-team empirical re-run — end-to-end against a **real bd v1.1.0 binary
with a real sqlite-backend workspace** (scratch harness
`rt_toctou_realbd_sqlite_e2e_test.go`, injected re-claim between the
staleness check and the release write; harness is scratch-only, not part
of this diff):

```
$ go test ./cmd/gc/ -run 'TestRT_RealBdSqlite' -count=1 -v
```

- **Unpatched base (`faa1d5bd1` + harness): 1/3 FAIL.**
`TestRT_RealBdSqlite_ReclaimBetweenCheckAndWriteNotClobbered`: `released
= [{gc-p6n 0}], want none when a concurrent claim landed mid-release` —
the orphan release discarded the live `worker-live` re-claim.
- **Patched: 3/3 PASS.** The injected mid-window re-claim is preserved
(`in_progress/worker-live`) and skipped loudly
(`releaseOrphanedPoolAssignments: skipping release for gc-5sa:
assignment changed between staleness check and release write`);
`ReleaseIfCurrent` on sqlite surfaces `ErrConditionalReleaseUnsupported`
(not a stranding error); and the regression guard confirms an
uncontended sqlite orphan is still released, not permanently skipped.

Pre-commit (lint `0 issues`, `go vet ./...`, doc-gen) and the pre-push
full fast suite both passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant