Skip to content

fix(sentinel): per-pod Service must use SingleStack, not PreferDualStack - #97

Merged
chideat merged 1 commit into
chideat:mainfrom
kundulun:fix/sentinel-service-singlestack
Jul 13, 2026
Merged

fix(sentinel): per-pod Service must use SingleStack, not PreferDualStack#97
chideat merged 1 commit into
chideat:mainfrom
kundulun:fix/sentinel-service-singlestack

Conversation

@kundulun

Copy link
Copy Markdown
Contributor

On a dual-stack cluster the sentinel per-pod Service traps the operator in an infinite reconcile loop, which in turn causes spurious Valkey failovers.

The bug

GeneratePodNodePortService (internal/builder/sentinelbuilder/service.go#L110) sets:

IPFamilies:     protocol,                                        // single family: [IPv4] or [IPv6]
IPFamilyPolicy: ptr.To(corev1.IPFamilyPolicyPreferDualStack),    // <-- asks for dual-stack

These two contradict each other. On a dual-stack cluster the apiserver honours PreferDualStack and assigns both families to the Service. On the next reconcile, util.IsServiceChanged compares the live Service against the desired single-family list, sees a mismatch, and issues an update that tries to release the secondary family without switching the policy to SingleStack. The apiserver rejects it:

Service "rfs-<name>-2" is invalid: spec.ipFamilyPolicy: Invalid value:
"PreferDualStack": must be 'SingleStack' to release the secondary IP family

Per the Kubernetes dual-stack docs, releasing a secondary IP family requires ipFamilyPolicy: SingleStack.

The condition is unrecoverable, and since the error is not returned from Reconcile (it is wrapped into a requeue result), there is no exponential backoff — the controller requeues every 15s, forever. The error also surfaces permanently in Sentinel.status.message.

Why it matters beyond the noise

Master liveness is inspected on every reconcile (engine.InspectisNodesHealthy). The hot requeue loop therefore hammers the master check continuously, and any transient inability to reach the master (slow node, brief I/O stall) makes the operator declare a perfectly healthy master dead and force SENTINEL FAILOVER — while the sentinels themselves never flagged it (+sdown/+odown absent).

In our cluster (5 Valkey instances, dual-stack) this produced ~20 spurious master failovers per day, and the reconcile loop had been running for 42–62 days across all instances.

The fix

One line: use SingleStack, consistent with every other service builder in the repoclusterbuilder, failoverbuilder, and the sentinel headless service in this very same file (L46/L64) all use SingleStack. This line is the odd one out; it was changed in ce86c86 ("feat: code clean").

Note: the alternative (keep PreferDualStack and list both families) would be a behavioural change; SingleStack restores the original, consistent behaviour.

Verification

Deployed a patched image to our production cluster:

  • reconcile error loop: ~25 errors/min → 0
  • all sentinel per-pod Services converged to SingleStack/[IPv4] cleanly (the update is accepted, the secondary family is released)
  • Sentinel.status went from a permanent error message to Ready
  • no data-plane disruption: valkey pods untouched, masters retained

Related: #91 addresses the second half of the problem (operator declaring a reachable-but-slow master dead); this PR removes the amplifier that makes it fire so often.

GeneratePodNodePortService sets IPFamilyPolicy: PreferDualStack while
IPFamilies is a single-family slice ([IPv4] or [IPv6]). On a dual-stack
cluster the apiserver then assigns BOTH families to the Service. On the
next reconcile IsServiceChanged detects the mismatch against the desired
single-family list and issues an update that tries to release the
secondary family without switching the policy, which the apiserver
rejects:

  Service "rfs-xxx-2" is invalid: spec.ipFamilyPolicy: Invalid value:
  "PreferDualStack": must be 'SingleStack' to release the secondary IP
  family

Per the Kubernetes dual-stack docs, releasing a secondary IP family
requires ipFamilyPolicy to be SingleStack. The error is unrecoverable, so
the sentinel reconcile requeues forever (the error is not returned from
Reconcile, so there is no exponential backoff).

Side effect: because master liveness is inspected on every reconcile, the
hot requeue loop greatly increases the chance of a spurious failover when
a master is briefly unreachable.

Every other service builder in this repo (clusterbuilder, failoverbuilder,
and the sentinel headless service in this very file) uses SingleStack —
this line is the odd one out. Introduced in ce86c86 (feat: code clean).

Refs: https://kubernetes.io/docs/concepts/services-networking/dual-stack/
@chideat

chideat commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Thanks a lot for this, @kundulun — excellent work. 🙏

This is a textbook bug report: you traced the contradiction between the single-family IPFamilies list and the PreferDualStack policy all the way through IsServiceChanged to the "must be 'SingleStack' to release the secondary IP family" rejection and the resulting 15s hot-requeue loop, pinpointed the regression to ce86c86, and even verified the fix (and the downstream failover amplification) on a real dual-stack production cluster. The write-up made review easy.

I confirmed the diagnosis independently: this line is the lone outlier — the sentinel headless service in the same file, plus every service builder in clusterbuilder and failoverbuilder (including the sibling GeneratePodNodePortService there), all use SingleStack. The change restores the original, consistent behavior with zero data-plane impact.

Merging now. As a small follow-up (non-blocking), it'd be great to add a sentinelbuilder service test asserting SingleStack, mirroring the existing clusterbuilder/service_test.go assertions — that would have caught this at CI time and guards against re-regression. Thanks again! 🎉

@chideat chideat left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and verified independently. The fix correctly aligns the sentinel per-pod Service with SingleStack — matching the headless service in the same file plus every builder in clusterbuilder/failoverbuilder — and restores the pre-ce86c86 behavior. CI (Test with Coverage) is green and there are no conflicts. LGTM.

@chideat
chideat merged commit c44fdf1 into chideat:main Jul 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants