test(flagd): run the provider conformance suite against both resolvers

[aepfli]

Stacked on #409. Base is feat/provider-tck, so this diff shows only the new work. Review #409 first.

Part of open-feature/spec#417 and #410.

Draft: the flagd suites have never been run. No Docker and no checked-out flagd submodules on the machine this was written on. Every capability declaration was derived by reading the resolver that would have to satisfy it, and every provider keyword against provider.py's signature, but CI is the first execution. The Go equivalent (go-sdk-contrib#941) shipped under the same constraint for the same reason.

What

Adopts the conformance suite in the flagd provider, for both resolvers, as two separate suites.

They are separate because they are separately conformant. flagd resolves flags two quite different ways — RPC evaluates remotely over gRPC, in-process syncs the ruleset and evaluates locally with openfeature-flagd-core — and any difference between the two results is a difference an application would see when it switches resolver. That is exactly the class of thing the suite exists to surface.

flagd-testbed is not modified, and the existing flagd e2e suites are untouched. The TCK drives the testbed's launchpad through the standardised control API, which the launchpad already implements, and reuses the container lifecycle already in tests/e2e.

The adoption

Two modules over one shared build_config, plus the per-resolver configuration. Each declares its capability set and calls pytest-bdd's scenarios(features_path()) directly — the step definitions arrive through the TCK's pytest11 entry point, so tests/tck/ needs no conftest.py for them, only for the container and control fixtures.

The stack is started once per session and never restarted. Scenario isolation comes from the control API instead, because container orchestrators assign host ports dynamically and cannot reliably preserve them across a restart — a restarted backend comes back on a different host port, silently invalidating every provider already pointed at the old one, and the failure looks like a flaky provider rather than a broken test.

One stack serves both suites, because one flagd process serves both ports the resolvers use (8013 RPC, 8015 sync), so there is nothing a second stack would isolate. The HttpControl is shared for a less obvious reason: it tracks whether a disconnect has left the backend down, so the next scenario starts it rather than merely resetting flag state. Two instances would each hold half of that knowledge.

The @unavailable provider points at a port discovered by binding and releasing, never at the backend under test — that has to stay up, and simulated outages belong to the control API.

HttpControl

The client for the control API in control-api.yaml, added to the TCK package in this PR because nothing in #409 needed it.

It is urllib.request only, so the TCK gains no new dependency — not an HTTP client, and not a container runtime. Orchestrating the stack stays with the adopter, where the vendor-specific knowledge already lives (which compose file, which services, which internal ports). That is a deliberate trade against the "provider authors write no test infrastructure" goal, and worth revisiting once a second containerised adopter shows what is actually common; abstracting from one example tends to produce the wrong abstraction.

Two behaviours worth reviewing:

• /reset fallback. prepare_scenario prefers POST /reset, which restores the baseline with no availability blip and so cannot inject a spurious lifecycle event into the next scenario. It is optional; flagd-testbed's launchpad registers only /start, /restart, /stop and /change, so it answers 404 and the TCK falls back to POST /start?config=default. The fallback is the normal path here, not the exception. The probe happens once per session and the answer is cached in _reset_supported.
• After a disconnect, /start rather than /reset. /reset is specified to reset flag state, not to start a stopped backend, so HttpControl tracks whether a disconnect happened and starts the backend for the scenario that follows one.

Finding: Python's RPC resolver does emit PROVIDER_STALE, and Go's does not

The Go adoption withholds Capability.STALE from its RPC suite, because Go's RPC resolver sends ProviderError directly on connection loss and never emits PROVIDER_STALE — filed as go-sdk-contrib#939.

Python has no such asymmetry. resolvers/grpc.py:202-212: the channel-connectivity callback emits PROVIDER_STALE on TRANSIENT_FAILURE and only then starts a threading.Timer that escalates to PROVIDER_ERROR once retry_grace_period expires. That is structurally the same staged handling as the in-process watcher at grpc_watcher.py:178-190.

	Python RPC	Python in-process	Go RPC
PROVIDER_STALE on connection loss	yes	yes	never
PROVIDER_ERROR	after grace period	after grace period	immediately

So both Python suites declare the full capability set, including STALE.

This is not a point scored. It is a cross-language behavioural difference in the same provider, against the same specification, and the conformance suite is what made it visible — which is roughly the argument for having one. Whichever way it gets reconciled, it should be reconciled deliberately rather than by accident of implementation.

No xfail for the bool-satisfies-Integer bug

The Python SDK narrows bool to Integer in typed evaluation (python-sdk#619), which would ordinarily put the STRICT_NUMERIC_TYPING scenarios at risk. It does not affect flagd, because flagd errors first, before the SDK's check is reached:

• RPC does not type-check locally. It asks flagd for an Int, the server answers INVALID_ARGUMENT, and grpc.py:461-462 maps that to TypeMismatchError.
• In-process rejects it explicitly: flagd_core.py:222-223, "For integer type, reject bool (since bool is subclass of int)", raising TypeMismatchError.

So both resolvers declare STRICT_NUMERIC_TYPING unconditionally and no xfail is needed. If #619 is fixed, nothing here changes.

CI

The paths filter already has an entry for providers/openfeature-provider-flagd, and #409 added one for tools/openfeature-provider-tck, so no workflow change is needed.

But the build workflow will not run on this PR while it is stacked. .github/workflows/build.yml triggers on pull_request: branches: [main], and this PR targets feat/provider-tck. It gets checks once #409 merges and GitHub retargets this PR to main. (lint-pr.yml and component-owners.yml use pull_request_target with no branch filter, so those two do run — a PR-title check and an owner ping, neither of which builds or tests anything.)

Worth stating plainly, because it means the flagd suites in this PR have not been executed by anything — not locally, not by CI. The TCK side is verified; see below.

uv.lock is updated in its own commit. The flagd dev group now depends on openfeature-provider-tck as a workspace source, and without the lock entry uv sync --frozen fails the build job for every package in the workspace, not just flagd.

Verification

Check	Result
ruff format --check, both trees	pass — 25 files already formatted
ruff check, both trees	pass
pytest tests in tools/openfeature-provider-tck	pass — 73 passed, 7 skipped, 2 xfailed
uv lock --check	pass — the regenerated lock is a two-line diff
flagd conformance suites (tests/tck)	not run — no Docker, flagd submodules not checked out
mypy tools/openfeature-provider-tck	2 errors, both pre-existing on feat/provider-tck and unrelated to this PR — set[Capability] passed where frozenset[Capability] is expected, in test_in_memory_conformance.py:93 and test_controllable_conformance.py:42. This branch adds two source files and no new errors. Belongs on #409.
FlagdProvider keywords used by suite.py	checked against provider.py:46-69 — resolver_type, host, port, deadline_ms, stream_deadline_ms, retry_backoff_ms, retry_backoff_max_ms, retry_grace_period all exist
FlagdContainer API	checked against tests/e2e/flagd_container.py — get_port(ResolverType), get_launchpad_url(), start(), stop()
Control-API verbs vs flagd-testbed	checked against the launchpad — /start?config=, /stop, /restart?seconds=, /change; no /reset, hence the fallback
Capability declarations	derived by reading both resolvers' event emission, line by line, not by running the suite

The capability sets are the part most likely to need adjusting after the first real run. If a declared capability fails, the fix is to withhold it and file an issue against the provider — not to weaken the scenario.

Worth filing

Two things the reading turned up that are provider bugs rather than suite problems, and are deliberately not fixed here:

1. In-process emits PROVIDER_CONFIGURATION_CHANGED on every reconnect, unconditionally. in_process.py:30-36 — _FlagStoreAdapter.update() calls emit_provider_configuration_changed for every sync payload the watcher applies, including one whose changed_keys is empty. A reconnect that re-delivers an unchanged ruleset therefore announces a configuration change that did not happen. Handlers that invalidate caches or re-render on that event do unnecessary work, and an application cannot distinguish a real change from a reconnect.

2. RPC passes through STALE before ERROR against a backend that was never reachable. The staged handling described above is right for losing a connection, but it also fires when the initial connection never succeeded — STALE means "cannot reach the backend but still serving values", and there are no values to serve. This is the likeliest source of @unavailable flakiness on the first real run: the scenario expects a prompt PROVIDER_ERROR, and the resolver will insert a retry_grace_period-long STALE window ahead of it. The suites keep short deadlines and a 1s grace period for the unavailable provider precisely to bound this, but bounding it is not the same as fixing it.

Not filed as issues from here — flagging them so someone with context on the flagd provider can decide.

What is not here

• flagd's file resolver. It has no backend to disconnect from, so it would use InProcessControl and a different capability set. Left out to keep the reviewable surface small.
• A shared containerised-backend helper. See the HttpControl section.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}