[BUG] flagd-proxy: subscribing to a not-yet-existing FeatureFlag permanently wedges that target until restart

[yuchou87]

Observed behavior

If a client subscribes to a FeatureFlag CR that does not exist yet, and the CR is
created afterwards, the client never receives the flag data. flagd-proxy stops
logging anything at all for that target — not even starting sync — even though the
client keeps reconnecting every 1–12s.

Restarting the client does not help. Only restarting flagd-proxy does, and
recovery is then immediate (the client does not even need to reconnect deliberately).

Since flagd-proxy is a cluster-wide singleton, the only known recovery affects every
service in the cluster.

Why this matters

This is the normal onboarding order, not an edge case:

1. A service integrates the SDK and ships. It is not using any flag yet, so no CR
   exists.
2. Later, someone enables the first flag → the control plane creates the CR.
3. Running pods are expected to pick it up within seconds.

Step 3 never happens. Dynamic flag delivery degrades into "restart the shared proxy".

This also explains why it stayed hidden for us: every service we had onboarded so far
was created CR-first, so none of them had ever taken the "service exists before the
CR" path.

Evidence

While the CR was missing (repeated every 1–12s, dozens of times over ~3 minutes):

starting sync from my-ns/my-flags
error with the initial fetch: unable to fetch FeatureFlag my-ns/my-flags:
  featureflags.core.openfeature.dev "my-flags" not found
manager.go:212 error from sync for target core.openfeature.dev/my-ns/my-flags: ...

The CR was then created, with complete content.

After that — nothing for that target, for 20+ minutes, while other targets kept
emitting kube sync notifier event every 30–60s. So the proxy process itself was
healthy.

A client pod started 9 minutes after the CR existed still timed out:

Provider flagd transitioned from state NOT_READY to state ERROR
Initialization timeout exceeded; did not complete within the 10000 ms deadline

and produced no proxy-side log line at all — no starting sync, no
unable to initiate sync.

Client and network ruled out from inside that pod: DNS resolved, TCP to
flagd-proxy-svc:8015 connected, and evaluation returned FLAG_NOT_FOUND rather than
PROVIDER_NOT_READY — i.e. the provider was serving, its store was simply empty.

After rollout restart deploy/flagd-proxy, without touching the client:

starting sync from my-ns/my-flags        # no error this time
kube sync notifier event: add: my-ns my-flags

The client had been retrying the whole time, so what was wedged was the proxy's
subscription state, not the client.

Root cause

flagd-proxy/pkg/service/subscriptions/manager.go.

RegisterSubscription decides purely on map membership and never checks whether the
multiplexer is still alive:

sh, ok := s.multiplexers[target]
if !ok {
    s.multiplexers[target] = &multiplexer{...}
    go s.watchResource(target)                 // watcher started only when absent
} else {
    sh.subs[key] = storedChannels{...}         // attach only; no watcher restarted
    if sh.syncRef != nil {
        go func() {
            s.mu.RLock()
            defer s.mu.RUnlock()
            if _, ok := s.multiplexers[target]; ok {   // re-checks membership, not liveness
                ...sh.syncRef.ReSync(ctx, dataSync)...
            }
        }()
    }
}

The else branch does attempt a ReSync, which is what would otherwise rescue the new
subscriber. But that attempt is guarded by a second membership check, and it runs on its
own goroutine — so when the delete lands in between, the check fails and the ReSync is
skipped. Neither guard tests whether the multiplexer is still being watched.

Meanwhile watchResource removes the entry asynchronously on ctx cancellation:

ctx, cancel := context.WithCancel(s.ctx)
defer cancel()
...
go func() {
    <-ctx.Done()
    s.mu.Lock()
    delete(s.multiplexers, target)       // deleted on another goroutine
    s.mu.Unlock()
}()
...
err = syncSource.Sync(ctx, sh.dataSync)
if err != nil {
    s.logger.Error(...)
    sh.broadcastError(s.logger, err)
}
// function returns → deferred cancel() → only then does the goroutine delete

A subscription arriving between watchResource returning and that goroutine deleting
the entry takes the else branch and attaches to a multiplexer whose context is
already cancelled and whose broadcast goroutine has exited. It can never receive data,
and nothing will restart a watcher for it.

With the client retrying every 1–12s and dozens of failures while the CR was missing,
landing inside that window is close to inevitable.

The trigger is probably not specific to a missing CR

A missing CR is how we hit this, but reading the code, the window opens whenever
watchResource returns — which is after any Sync error, including transient ones
such as an API server blip or a dropped watch connection. There is also a self-feeding
loop: the error is broadcast to the subscribers, the client reconnects promptly, and
that reconnect lands in the window that the same error just opened. A missing CR is
simply the variant that fails over and over, giving many chances to land there.

We have only observed the missing-CR case in production, so treat the wider framing as a
code-level inference — though the reproduction below does support it, since it wedges
using an ordinary error rather than anything Kubernetes-specific.

This distinction matters for the fix: if the trigger really is only the onboarding
order, closing the window (the first option below) is enough; if any transient sync error
can do it, the liveness check (the second option) is the one that actually holds.

Reproduced in a unit test

The interleaving above is not just inferred — it reproduces against main (e045237)
using the mocks already in flagd-proxy/pkg/service/subscriptions/manager_test.go, with
no cluster involved. syncMock.errChanIn stands in for "the FeatureFlag does not exist",
and broadcastError runs before watchResource returns, so observing the error puts
the second subscription inside the window by construction rather than by luck:

1. RegisterSubscription(k1) → !ok branch → watchResource starts; confirm it is live
   by pushing one DataSync through.
2. Send an error on errChanIn → Sync returns → watchResource broadcasts and is
   about to return.
3. RegisterSubscription(k2) → else branch.
4. Push another DataSync. k2 never receives it.

Debug logging confirms each step:

manager.go:112  sync handler does not exist for target ns/flags, registering multiplexer with sub k1
manager.go:163  watching resource ns/flags
manager.go:212  error from sync for target ns/flags: featureflags.core.openfeature.dev "flags" not found
                multiplexer still present in the map: true
manager.go:131  registering sync subscription k2          <- else branch, no watcher restarted
                k2 received nothing

Note what is absent: there is no triggering a resync line. The ReSync goroutine ran
after the delete had landed, so its membership re-check failed and it returned without
doing anything. Both recovery paths miss.

The timing is what decides it, and both outcomes reproduce:

when the second subscription arrives	branch	result
entry still present (inside the window)	else	wedged — no watcher, and the ReSync is skipped
entry already deleted	!ok	recovers — a new multiplexer and watcher are built

Run as a loop without deliberately timing step 3, 14 of 20 subscriptions were wedged.

Note that manager.go:212 is the same line as in the production log quoted above.

We are happy to contribute this as a regression test alongside a fix.

What is and is not established

Confirmed: the symptom reproduces in a real cluster; only a proxy restart recovers
it; client and network are healthy; the CR content is complete; the proxy is silent for
the affected target while healthy for others. The interleaving described above is also
confirmed — directly observed in the unit-test reproduction, not inferred.

Not explained: why the window stayed open for so long in production. A client pod
starting 9 minutes after the CR existed still took the else branch, which means the
entry was still in the map at that point — long after the failing watchResource had
returned and its deferred cancel() should have triggered the delete. The unit test
reproduces the wedge within the normal, short window; it does not reproduce a window
that stays open for minutes. Something else may be holding the entry alive, and the fix
below should be judged on its own merits rather than on this being fully understood.

Impact

Evaluation availability	No crash, but every flag for that service resolves to its built-in default until the proxy restarts
Direction of failure	Safe — flags stay off rather than on
Dynamic delivery	Completely broken, which is the point of a flag system
Blast radius of the workaround	Restarting a cluster-wide singleton affects every service in the cluster

How to tell it is this

• Client side: initialization timeout, and evaluation returns FLAG_NOT_FOUND rather
  than PROVIDER_NOT_READY
• Proxy side: no log lines at all for that target, while other targets keep
  emitting notifier event normally

Workaround

Create the FeatureFlag CR before the service starts — even an empty one. That way
the resource already exists at first subscription and the bad state is never entered.

For an instance already wedged, restart flagd-proxy.

Expected Behavior

A subscription to a FeatureFlag that does not exist yet should start delivering data
once the CR is created, without operator intervention. A target should never be left in
a state where no watcher is running and nothing will ever start one — and if it is,
recovering it should not require restarting a cluster-wide singleton.

Proposed fix

Either make the removal synchronous, closing the window:

defer func() {
    s.mu.Lock()
    delete(s.multiplexers, target)
    s.mu.Unlock()
    cancel()
}()

or, more robustly, validate liveness before attaching in the else branch and rebuild
the multiplexer when it is dead:

sh, ok := s.multiplexers[target]
if !ok || sh.isDead() {
    // rebuild + go s.watchResource(target)
}

The second also covers a stale entry left behind for any other reason, and does not
depend on pinning down the exact interleaving. The first is smaller and more direct.
Ideally both. Happy to open a PR.

Steps to reproduce

There are two ways to reproduce. The first needs nothing but go test.

Without a cluster

Described in full under "Reproduced in a unit test" above: drive
RegisterSubscription directly with the mocks already in
flagd-proxy/pkg/service/subscriptions/manager_test.go, fail the sync via
syncMock.errChanIn, and register a second subscription while the entry is still in the
map. No Kubernetes, no client SDK, no provider. This is how we confirmed the
interleaving, and we are happy to contribute it as a regression test.

In a cluster (how we originally hit it)

1. Deploy flagd-proxy.
2. Point an in-process provider at core.openfeature.dev/<ns>/<name> where that
   FeatureFlag does not exist. Let it retry for a minute or two.
3. kubectl apply the FeatureFlag.
4. Observe: the client still times out; flagd-proxy logs nothing for that target.
5. kubectl rollout restart deploy/flagd-proxy → the target syncs immediately.

Environment for the cluster reproduction

The unit-test route above needs none of this; these are the versions we observed the
problem on.

• flagd-proxy v0.9.8 — this is not fixed by the two defects resolved in that
  release; it was found while verifying those fixes
• open-feature-operator v0.9.2 (its default proxy image is v0.9.4; tag overridden
  to v0.9.8)
• Kubernetes, managed cloud offering
• Client: OpenFeature Java SDK 1.22.0 + flagd provider 0.14.0, in-process resolver