fix(sentinel): per-pod Service must use SingleStack, not PreferDualStack - #97
Conversation
GeneratePodNodePortService sets IPFamilyPolicy: PreferDualStack while IPFamilies is a single-family slice ([IPv4] or [IPv6]). On a dual-stack cluster the apiserver then assigns BOTH families to the Service. On the next reconcile IsServiceChanged detects the mismatch against the desired single-family list and issues an update that tries to release the secondary family without switching the policy, which the apiserver rejects: Service "rfs-xxx-2" is invalid: spec.ipFamilyPolicy: Invalid value: "PreferDualStack": must be 'SingleStack' to release the secondary IP family Per the Kubernetes dual-stack docs, releasing a secondary IP family requires ipFamilyPolicy to be SingleStack. The error is unrecoverable, so the sentinel reconcile requeues forever (the error is not returned from Reconcile, so there is no exponential backoff). Side effect: because master liveness is inspected on every reconcile, the hot requeue loop greatly increases the chance of a spurious failover when a master is briefly unreachable. Every other service builder in this repo (clusterbuilder, failoverbuilder, and the sentinel headless service in this very file) uses SingleStack — this line is the odd one out. Introduced in ce86c86 (feat: code clean). Refs: https://kubernetes.io/docs/concepts/services-networking/dual-stack/
|
Thanks a lot for this, @kundulun — excellent work. 🙏 This is a textbook bug report: you traced the contradiction between the single-family I confirmed the diagnosis independently: this line is the lone outlier — the sentinel headless service in the same file, plus every service builder in Merging now. As a small follow-up (non-blocking), it'd be great to add a |
chideat
left a comment
There was a problem hiding this comment.
Reviewed and verified independently. The fix correctly aligns the sentinel per-pod Service with SingleStack — matching the headless service in the same file plus every builder in clusterbuilder/failoverbuilder — and restores the pre-ce86c86 behavior. CI (Test with Coverage) is green and there are no conflicts. LGTM.
On a dual-stack cluster the sentinel per-pod Service traps the operator in an infinite reconcile loop, which in turn causes spurious Valkey failovers.
The bug
GeneratePodNodePortService(internal/builder/sentinelbuilder/service.go#L110) sets:These two contradict each other. On a dual-stack cluster the apiserver honours
PreferDualStackand assigns both families to the Service. On the next reconcile,util.IsServiceChangedcompares the live Service against the desired single-family list, sees a mismatch, and issues an update that tries to release the secondary family without switching the policy toSingleStack. The apiserver rejects it:Per the Kubernetes dual-stack docs, releasing a secondary IP family requires
ipFamilyPolicy: SingleStack.The condition is unrecoverable, and since the error is not returned from
Reconcile(it is wrapped into a requeue result), there is no exponential backoff — the controller requeues every 15s, forever. The error also surfaces permanently inSentinel.status.message.Why it matters beyond the noise
Master liveness is inspected on every reconcile (
engine.Inspect→isNodesHealthy). The hot requeue loop therefore hammers the master check continuously, and any transient inability to reach the master (slow node, brief I/O stall) makes the operator declare a perfectly healthy master dead and forceSENTINEL FAILOVER— while the sentinels themselves never flagged it (+sdown/+odownabsent).In our cluster (5 Valkey instances, dual-stack) this produced ~20 spurious master failovers per day, and the reconcile loop had been running for 42–62 days across all instances.
The fix
One line: use
SingleStack, consistent with every other service builder in the repo —clusterbuilder,failoverbuilder, and the sentinel headless service in this very same file (L46/L64) all useSingleStack. This line is the odd one out; it was changed in ce86c86 ("feat: code clean").Note: the alternative (keep
PreferDualStackand list both families) would be a behavioural change;SingleStackrestores the original, consistent behaviour.Verification
Deployed a patched image to our production cluster:
SingleStack/[IPv4]cleanly (the update is accepted, the secondary family is released)Sentinel.statuswent from a permanent error message toReadyRelated: #91 addresses the second half of the problem (operator declaring a reachable-but-slow master dead); this PR removes the amplifier that makes it fire so often.