Skip to content

fix: delete per-session prometheus gauge series on Syncer.Close#22

Closed
mxssl wants to merge 1 commit into
mainfrom
fix/session-gauge-cardinality-leak
Closed

fix: delete per-session prometheus gauge series on Syncer.Close#22
mxssl wants to merge 1 commit into
mainfrom
fix/session-gauge-cardinality-leak

Conversation

@mxssl

@mxssl mxssl commented Jul 5, 2026

Copy link
Copy Markdown

Problem

The gauges chotki_sync_sessions (labels id, kind, version), chotki_sync_opened_snapshots and chotki_sync_opened_iterators (labels id, version) use the connection name as the id label. Listen-side names (listen:<uuid>:<ip>:<port>) are unique per connection, and the series were never removed when a session terminated — they accumulate until the process restarts.

Observed on a long-running production deployment (2026-07-05): 84,320 stale chotki_sync_sessions series in state 4 (SendNone) vs 135 live sessions; long-lived nodes exposed ~46k samples per /metrics scrape vs ~10k on freshly-restarted ones. Unbounded label cardinality like this eventually pushes /metrics past scraper response-size limits, at which point the entire scrape gets dropped silently.

Fix

Syncer.Close() now deletes the per-session series. network.Peer.Close() calls it only after wg.Wait() has joined the read/write loops, so no Feed/Drain runs concurrently with the deletion (and a straggler double-Close is self-cleaning because the deletes are deferred to function exit).

One subtlety: alerting on leaked pebble resources relies on opened_snapshots/opened_iterators staying nonzero when a close fails. Deleting those series unconditionally would blind it. So:

  • chotki_sync_sessions (feed + drain) series are always deleted, including the nil-Host early-return path.
  • Snapshot/iterator series are deleted only when every close succeeded. Mid-session close failures in Feed() (SendDiff/SendEOF paths) and FeedDiffVV() nil the handles while leaving the gauge at 1, so new snapCloseFailed/iterCloseFailed flags (set under the lock) tell Close() to keep those series as the leak signal.

Testing

New white-box tests in replication/sync_metrics_test.go: clean close removes all per-session series; close with a failing snapshot keeps the snapshot series; mid-session close failure (flags set) keeps snapshot+iterator series; nil-Host close still removes state series. Full go test ./... passes.

Review notes

Both reviewers noted that Feed/FeedDiffVV mutate snap/ffit/vvit outside sync.lock while Close() reads them under it — that is pre-existing behavior, safe under the current Peer.Close ordering guarantee, and left untouched to keep this change minimal. Related follow-up candidate (not in this PR): chotki_outbound_packet_count (CounterVec, per-peer name label) leaks series the same way.

chotki_sync_sessions / chotki_sync_opened_snapshots / chotki_sync_opened_iterators
carry a unique per-connection id label but their series were never deleted when a
session ended, so they accumulate until process restart. Close() now drops the
state series unconditionally and the snapshot/iterator series only when the
underlying close succeeded (mid-session close failures set a flag), so the
leak-alert signal on stuck snapshots/iterators stays visible.
@mxssl
mxssl requested a review from a team as a code owner July 5, 2026 11:55
@mxssl
mxssl requested a review from msizov July 6, 2026 12:16
@mxssl

mxssl commented Jul 10, 2026

Copy link
Copy Markdown
Author

Closing this — the per-session prometheus gauge cleanup on Syncer.Close is already fixed in #21.

@mxssl mxssl closed this Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant