main.zig runs processor.runInitialSnapshot(...) before it spawns the observability worker, so /metrics, /healthz, and /readyz refuse connections until the snapshot has fully drained. obs.markConnected(true) and the first obs.heartbeat also live in processor.startStreaming, which only runs afterwards.
On a large table the snapshot runs for minutes. Under Kubernetes with an HTTP liveness or startup probe on /healthz the probe fails, the pod is killed mid-snapshot, and the restart finds the snapshot marker in place, drops the slot, and redoes the snapshot from scratch. With a table big enough to outlast the probe budget that never converges.
Before the initial snapshot landed there was no window: streaming started right after connect, so the server came up immediately.
Ask: start the observability worker before runInitialSnapshot and report readiness for the snapshot phase, so a slow bootstrap is visible instead of indistinguishable from a dead process.
main.zigrunsprocessor.runInitialSnapshot(...)before it spawns the observability worker, so/metrics,/healthz, and/readyzrefuse connections until the snapshot has fully drained.obs.markConnected(true)and the firstobs.heartbeatalso live inprocessor.startStreaming, which only runs afterwards.On a large table the snapshot runs for minutes. Under Kubernetes with an HTTP liveness or startup probe on
/healthzthe probe fails, the pod is killed mid-snapshot, and the restart finds the snapshot marker in place, drops the slot, and redoes the snapshot from scratch. With a table big enough to outlast the probe budget that never converges.Before the initial snapshot landed there was no window: streaming started right after connect, so the server came up immediately.
Ask: start the observability worker before
runInitialSnapshotand report readiness for the snapshot phase, so a slow bootstrap is visible instead of indistinguishable from a dead process.