fix: delete per-session prometheus gauge series on Syncer.Close#22
Closed
mxssl wants to merge 1 commit into
Closed
fix: delete per-session prometheus gauge series on Syncer.Close#22mxssl wants to merge 1 commit into
mxssl wants to merge 1 commit into
Conversation
chotki_sync_sessions / chotki_sync_opened_snapshots / chotki_sync_opened_iterators carry a unique per-connection id label but their series were never deleted when a session ended, so they accumulate until process restart. Close() now drops the state series unconditionally and the snapshot/iterator series only when the underlying close succeeded (mid-session close failures set a flag), so the leak-alert signal on stuck snapshots/iterators stays visible.
Author
|
Closing this — the per-session prometheus gauge cleanup on Syncer.Close is already fixed in #21. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The gauges
chotki_sync_sessions(labelsid,kind,version),chotki_sync_opened_snapshotsandchotki_sync_opened_iterators(labelsid,version) use the connection name as theidlabel. Listen-side names (listen:<uuid>:<ip>:<port>) are unique per connection, and the series were never removed when a session terminated — they accumulate until the process restarts.Observed on a long-running production deployment (2026-07-05): 84,320 stale
chotki_sync_sessionsseries in state 4 (SendNone) vs 135 live sessions; long-lived nodes exposed ~46k samples per/metricsscrape vs ~10k on freshly-restarted ones. Unbounded label cardinality like this eventually pushes/metricspast scraper response-size limits, at which point the entire scrape gets dropped silently.Fix
Syncer.Close()now deletes the per-session series.network.Peer.Close()calls it only afterwg.Wait()has joined the read/write loops, so noFeed/Drainruns concurrently with the deletion (and a straggler double-Closeis self-cleaning because the deletes are deferred to function exit).One subtlety: alerting on leaked pebble resources relies on
opened_snapshots/opened_iteratorsstaying nonzero when a close fails. Deleting those series unconditionally would blind it. So:chotki_sync_sessions(feed + drain) series are always deleted, including the nil-Host early-return path.Feed()(SendDiff/SendEOF paths) andFeedDiffVV()nil the handles while leaving the gauge at 1, so newsnapCloseFailed/iterCloseFailedflags (set under the lock) tellClose()to keep those series as the leak signal.Testing
New white-box tests in
replication/sync_metrics_test.go: clean close removes all per-session series; close with a failing snapshot keeps the snapshot series; mid-session close failure (flags set) keeps snapshot+iterator series; nil-Host close still removes state series. Fullgo test ./...passes.Review notes
Both reviewers noted that
Feed/FeedDiffVVmutatesnap/ffit/vvitoutsidesync.lockwhileClose()reads them under it — that is pre-existing behavior, safe under the currentPeer.Closeordering guarantee, and left untouched to keep this change minimal. Related follow-up candidate (not in this PR):chotki_outbound_packet_count(CounterVec, per-peernamelabel) leaks series the same way.