fix(reconciler): debounce pool scale_check before spawning new sessions - #6
Open
austinborn wants to merge 2 commits into
Open
fix(reconciler): debounce pool scale_check before spawning new sessions#6austinborn wants to merge 2 commits into
austinborn wants to merge 2 commits into
Conversation
Adds a process-scoped per-template scale_check confirm window to the
pool reconciler. Each tick records the raw count for every pool
template; new-demand consumers (ComputePoolDesiredStatesDebounced{,Traced})
see the minimum count across the last N=3 samples once the window is
full, and the raw count while it's still warming up. This collapses
the single-tick scale_check flap class of pathology while preserving
cold-start responsiveness and steady-state demand semantics.
Behavior:
* Templates whose probe failed this tick (poolScaleCheckPartialTemplates)
pass through the debouncer unchanged. Recording a probe failure
(count=0) into the window would force the next two ticks to 0 even
after a steady-state of 1, so the existing
retainScaleCheckPartialPoolDesired path is left to handle the
partial-failure session-preservation case.
* Window=3 is hard-coded to keep this PR small. A daemon config knob
can be layered on later if operators want to tune it.
* Tests pin window=1 in TestMain so existing single-call assertions
about spawn behavior continue to hold. Debouncer tests opt back in
via setPoolScaleCheckDebouncerWindowForTesting.
Motivation:
An operator observed builder pool slot 2 in a ~66s respawn loop —
scale_check_count flipped 0→1→0 in the reconciler trace even though
external `bd ready --metadata-field gc.routed_to=<pool> --unassigned`
probes always returned 0. Each transient 1 triggered an anonymous
new-tier spawn; the spawned session ran the work_query, found no
claimable bead, and drained. Debouncing the count across a small
window eliminates the cycle without changing the spawn-on-real-demand
contract.
Touch points:
* cmd/gc/pool_scale_check_debouncer.go (new) — singleton + window logic.
* cmd/gc/pool_desired_state.go — adds ComputePoolDesiredStatesDebounced
and ComputePoolDesiredStatesDebouncedTraced wrappers; pure entry
points are unchanged.
* cmd/gc/build_desired_state.go, city_runtime.go, cmd_start.go — swap
the reconciler call sites to the Debounced variants.
* cmd/gc/main_test.go — pin window=1 for tests.
* cmd/gc/pool_scale_check_debouncer_test.go (new) — cold-start,
sustained, single-tick flap, partial pass-through, per-template
isolation, and reset tests.
Generated by the operator's software factory.
City: factory-main · Agent: local-core.builder-2
On behalf of: @austinborn
Co-Authored-By: factory-bot <factory-bot@austinborn.invalid>
…sfied `h := append(d.history[template], count)` triggers gocritic's appendAssign because the resulting slice header is bound to a different variable than the input. The intent is identical when written as two statements — fetch the existing history into h, then append onto h — but the linter is happy. No behavior change. Generated by the operator's software factory. City: factory-main · Agent: local-core.builder-2 On behalf of: @austinborn Co-Authored-By: factory-bot <factory-bot@austinborn.invalid>
austinborn
pushed a commit
that referenced
this pull request
Jul 24, 2026
gastownhall#4151) ## What Makes the reconciler's pool orphan-release conditional on the claim it observed: prefer the store's atomic `ReleaseIfCurrent` CAS (status/assignee) and, where a store cannot conditionally release, re-verify the snapshot with a live read immediately before the write plus a verify-after read that logs a raced claim loudly — replacing the previous unconditional `Update(assignee:"", status:"open")`. ## Why The 2026-07-11 sqlite infra-store concurrency audit flagged this as defect #6 (MED): the controller's orphan path (`releaseOrphanedPoolAssignment`, `cmd/gc/pool_session_name.go`) was a check-then-act TOCTOU — snapshot assigned work, run staleness gates (plus a potentially slow detached probe), then write an unconditional release. A legitimate re-claim landing in that window (crash recovery racing a live or revived claimer) was silently clobbered back to open/unassigned, with exit 0 everywhere and no runtime signal. Compounding it, conditional release was dead on the sqlite infra backend: `bd sql` is unsupported there (empirically confirmed by the audit), so `BdStore.ReleaseIfCurrent` fail-loud dies — and the orphan path never attempted it anyway. Audit context: the same hammer ran ~5,900 real cross-process bd operations with 640/640 claim races producing exactly one winner — the engine-level claim CAS is sound. This gc-side unconditional write was the layer above it that could undo a legitimately won claim. A red-team harness against a real bd v1.1.0 sqlite workspace reproduced the clobber deterministically on unpatched code (numbers below). `internal/beads/bdstore.go` also gains a SEAM comment marking exactly where bd's native conditional-release verb slots in once it ships (callers already treat `ErrConditionalReleaseUnsupported` as "take the recheck fallback", so no caller changes will be needed). ## Verification TDD unit tests (5 new tests covering: CAS preferred when supported, lost CAS race means no unconditional retry, unsupported store rechecks before write, raced claim after the fallback write is logged loudly, uncontended orphan still releases): ``` $ go test ./cmd/gc/ -run 'TestReleaseOrphanedPoolAssignments|TestWorkAssignmentReleaseWorkBead' -count=1 ok github.com/gastownhall/gascity/cmd/gc 1.709s $ go vet ./cmd/gc/ ./internal/beads/ # clean ``` Red-team empirical re-run — end-to-end against a **real bd v1.1.0 binary with a real sqlite-backend workspace** (scratch harness `rt_toctou_realbd_sqlite_e2e_test.go`, injected re-claim between the staleness check and the release write; harness is scratch-only, not part of this diff): ``` $ go test ./cmd/gc/ -run 'TestRT_RealBdSqlite' -count=1 -v ``` - **Unpatched base (`faa1d5bd1` + harness): 1/3 FAIL.** `TestRT_RealBdSqlite_ReclaimBetweenCheckAndWriteNotClobbered`: `released = [{gc-p6n 0}], want none when a concurrent claim landed mid-release` — the orphan release discarded the live `worker-live` re-claim. - **Patched: 3/3 PASS.** The injected mid-window re-claim is preserved (`in_progress/worker-live`) and skipped loudly (`releaseOrphanedPoolAssignments: skipping release for gc-5sa: assignment changed between staleness check and release write`); `ReleaseIfCurrent` on sqlite surfaces `ErrConditionalReleaseUnsupported` (not a stranding error); and the regression guard confirms an uncontended sqlite orphan is still released, not permanently skipped. Pre-commit (lint `0 issues`, `go vet ./...`, doc-gen) and the pre-push full fast suite both passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[1,1,1]is1.poolScaleCheckPartialTemplates) are pass-through: a probe failure (raw count = 0) is not a real observation of demand and must not poison the window.retainScaleCheckPartialPoolDesiredcontinues to own session preservation across partial-failure ticks.Motivation
The pathology this addresses: an operator observed a pool slot in a ~66s respawn loop.
scale_check_countflipped0 ↔ 1in the reconciler trace, but external probes of the exact samebd ready --metadata-field gc.routed_to=<pool> --unassigned --exclude-type=epic --json --limit 0 | jq lengthalways returned 0. Each transient1raised new-tier demand, a session bead was created, the session ran the work_query, found no claimable bead, and drained. ~170 ghost spawns in 4h on a single slot.Whatever upstream cause briefly produced an unassigned routed bead inside one tick (most plausibly a release-and-restore round trip on the assignee within a single reconcile), the spawn loop is the symptom this PR collapses. The orphan-release path inside
releaseOrphanedPoolAssignmentsWhenSnapshotsCompletealready runsliveOpenSessionAssignmentExistsagainst the live session label set (the protection introduced in PR gastownhall#1856 / commite796e7d9), so pinning a release containing that fix wouldn't have changed behavior here — the loop has to be filtered downstream of demand observation.What changed
cmd/gc/pool_scale_check_debouncer.go(new) — singleton holding per-template ring buffer + thedebouncePoolScaleCheckCountsentry point. Mutex-protected; tests can opt out (window=1) or opt in (window=3+) viasetPoolScaleCheckDebouncerWindowForTesting.cmd/gc/pool_desired_state.go— addsComputePoolDesiredStatesDebouncedandComputePoolDesiredStatesDebouncedTracedwrappers. The pureComputePoolDesiredStates{,Traced}entry points are unchanged so existing tests of the compute kernel remain valid.cmd/gc/build_desired_state.go(1 site),cmd/gc/city_runtime.go(3 sites — beadReconcileTick fallback, controlDispatcherTick, loadDemandSnapshot),cmd/gc/cmd_start.go(1 site) — swap to the Debounced variants. These are every reconciler-driven path that consumesresult.ScaleCheckCounts.cmd/gc/main_test.go—setPoolScaleCheckDebouncerWindowForTesting(1)in TestMain so existing single-call spawn assertions don't have to seed history. Debouncer-specific tests opt back in viawithPoolScaleCheckDebouncerWindow(t, 3).cmd/gc/pool_scale_check_debouncer_test.go(new) — covers: cold-start pass-through, sustained demand, single-tick flap suppression (the canonical pathology), partial-template pass-through (and that partials don't poison the window), per-template isolation, window=1 is a no-op, and reset semantics.Trade-offs
N × tick_interval. At the typical ~10–30s tick cadence, that's 30–90s before sustained new demand confirms. In-flight new-tier sessions from prior ticks continue to count as spent demand viapoolInFlightNewRequestsduring this wait, so the slot doesn't sit idle.[daemon]config knob in a follow-up PR.Test plan
go test ./cmd/gc/ -run 'TestPoolScaleCheckDebouncer' -count=1— six new tests, all pass.go test ./cmd/gc/ -run 'TestComputePoolDesiredStates|TestComputeAwakeSet|TestRetainScaleCheckPartial|TestBuildDesiredState|TestSessionReconciler|TestReleaseOrphaned' -count=1— all pass.go test ./cmd/gc/... -count=1 -timeout 600s. Two failures (TestRunStartDriftCheck_RestartReturnsContinue,TestDoStartJSONAlreadyRunningSupervisorKeepsStdoutJSONOnly/drift_restart) reproduce identically on a cleanorigin/main(8028e93) — both expect/proc/<pid>/exeto be readable, which doesn't exist on macOS. Pre-existing macOS-specific, unrelated to this PR.