[Tracking] A cross-language conformance suite for OpenFeature providers

[aepfli]

The problem

OpenFeature's central promise is vendor neutrality: swapping providers must not change application
behaviour. Nothing currently verifies that.

Every provider tests differently. Some have thorough suites, some have almost none, and no two
agree on what "implements the provider contract" means. The consequences are real and already
visible:

• Providers disagree on whether a type mismatch returns the code default or throws.
• Providers disagree on whether an unreachable backend yields ERROR promptly, hangs, or throws
  out of provider registration.
• Providers disagree on which lifecycle events fire, and in what order, around a disconnect.
• Some providers silently coerce between numeric types instead of reporting TYPE_MISMATCH.

Every one of these is invisible until an application swaps providers in production. The spec
describes the correct behaviour in prose; there is no executable definition of it.

The proposal

A language-agnostic conformance suite for providers — a set of Gherkin feature files, a
standardised backend control API, and a canonical flag set — that each language's TCK implementation
packages and runs against real providers.

We have built a working proof of concept in Java, using flagd as the first provider under test. It
finds real bugs already (see Findings). This issue is the cross-language
tracking issue; it is opened for discussion before anything is merged into the spec repo.

What lives in this repository

Three artifacts, which must travel together:

Artifact	What it is
gherkin/*.feature	The canonical test cases — the executable definition of the provider contract
openapi/control-api.yaml	What a backend under test must expose so a TCK can drive it
flags/canonical-flags.json	The flag set those test cases assume

They belong here rather than in any single language's repository because:

1. They are the contract. If every language's TCK writes its own scenarios, we get N different
   definitions of conformance and are no better off than today.
2. Backend vendors implement the control API in their own language. flagd's testbed is Go. It
   should not have to read a Java repository to learn what POST /restart must do.
3. They cannot be split. A feature file that evaluates boolean-flag is meaningless without the
   flag definition; a disconnect scenario is meaningless without the endpoint that produces the
   disconnect. Splitting them across repositories guarantees drift.

Each language's TCK then vendors this repo (git submodule or equivalent) and packages the files into
its own distribution format, so adopting a TCK never requires a consumer to check out a submodule of
their own.

What lives in each language repository

The runner: step definitions, an abstract base test class owning the container lifecycle and control
API client, and whatever adoption ergonomics that language's testing culture expects.

Scope: the provider contract only

In scope — how a provider maps backend responses onto the spec's types and lifecycle:

• typed resolution details: value, variant, reason, error code
• keeping integer and float distinct, with no silent coercion
• error handling: type mismatch and unknown flag return the code default, report the right error
  code, and never throw
• lifecycle: reaching READY; settling into ERROR against an unreachable backend
• events: PROVIDER_READY, PROVIDER_ERROR, PROVIDER_STALE, PROVIDER_CONFIGURATION_CHANGED
• that a signalled configuration change is actually applied on re-evaluation

Out of scope — deliberately:

• backend evaluation logic, targeting and bucketing correctness. That is the backend's contract, not
  the provider's. Every flag in the canonical set resolves to its default variant.
• the provider↔backend wire protocol.
• SDK behaviour, which belongs to each SDK's own suite.

Two design decisions worth discussing

1. The backend under test is a Compose stack, not an image

The TCK does not start a container; it starts a docker-compose.yaml the vendor supplies. This lets
vendors compose a proxy, an edge service, or several containers — a single image cannot express
"backend behind a toxiproxy". The TCK only relies on conventions: a known service name, a known
container-internal port for the control API, and dynamically mapped external ports.

2. Containers are never stopped or restarted mid-suite

This is the most important normative requirement and the least obvious one:

Backend unavailability MUST be simulated inside the running stack — a process kill, a proxy toxic,
a socket block. Implementations MUST NOT stop, kill, remove or recreate a container while the
suite is running.

The reason is portability, not preference. Testcontainers cannot reliably preserve dynamically
mapped host ports across a container restart, and which bindings preserve them differs by
language. A TCK that restarts containers works in one language and mysteriously fails in another —
the worst possible property for a cross-language conformance suite.

flagd's testbed already does the right thing: it kills and restarts the flagd process inside a
container that keeps running. That is the reference behaviour.

The control API

Derived from the endpoints flagd's testbed "launchpad" already implements, so flagd conforms today
without modification.

Endpoint	Status	Semantics
POST /start?config={name}	required	start the backend, seed flags to that config's baseline
POST /stop	required	make the backend unreachable; must not stop the container
POST /restart?seconds={n}	required	bounded outage; flag state preserved across it
POST /change	required	change changing-flag's resolved value without a restart
POST /reset	optional	restore baseline with no outage; falls back to /start
GET /healthz	optional	readiness; falls back to a TCP port check

A second normative rule, which the PoC depends on for scenario isolation:

/start resets flag state to the named configuration's baseline. /restart preserves it. An
outage must be observable as a change in availability, never as a change in flag values.

The two optional endpoints exist so that flagd — which implements neither — can adopt the suite
unchanged, while giving vendors building a new testbed something better to aim at.

Capability tags

Not every provider implements every optional part of the spec. Scenarios exercising an optional
capability carry a tag; a provider declares what it supports and the rest are reported as
skipped, with the reason printed — never silently passed. A conformance suite that quietly goes
green on scenarios it did not run is worse than no suite.

Current tags: @events, @stale, @configuration-change, @object, @unavailable,
@strict-numeric-typing, plus reserved @targeting and @caching.

Findings from the PoC

Running the PoC against the flagd Java provider, first attempt, already surfaced a real defect:

flagd silently narrows a float flag to an integer. Evaluating float-flag (0.5) through the
integer API returns 0 with no error code at all, rather than TYPE_MISMATCH with the code
default. The application sees a plausible value and no indication anything went wrong — the worst
failure mode a feature flag can have. This is precisely the class of bug that only a shared,
adversarial suite finds, since the provider's own tests encode the provider's own assumptions.

That single finding is, we think, the argument for this issue.

Open questions for discussion

1. Directory layout in this repo. provider-tck/{gherkin,flags,openapi}/? Something else?

2. Versioning. Should the suite be versioned independently of the spec, so a provider can claim
   "conformant to provider-tck 1.2" without tracking spec revisions? Related: how do we add
   scenarios without instantly breaking every adopter's build?

3. The canonical flag set's format. The PoC expresses it in flagd's flag-definition format,
   because that is the only widely implemented vendor-neutral format today. Is that acceptable as a
   description that vendors translate, or do we want a neutral schema?

4. Is @strict-numeric-typing a capability or a hard requirement? It is modelled as a capability
   so providers with the defect can adopt today and see it reported as an explicit skip. That is
   pragmatic, but it does put a spec violation on the same footing as an optional feature. Better
   idea welcome — perhaps a separate "known deviations" concept.

5. Context passthrough. The suite cannot currently assert an evaluation context reached the
   backend. That needs an echo operation (GET /last-evaluation?) on the control API. Worth adding
   before the first stable release?

6. Reason values. The PoC asserts STATIC for flags with no targeting rules. Is that
   universally correct across providers, or does it need to be a per-provider expectation?

7. Is @strict-numeric-typing meaningful in every language? JavaScript has no integer type:
   typeof 10 and typeof 0.5 are both 'number' and the Evaluation API exposes only
   getNumberDetails, so requesting float-flag as an Integer is indistinguishable from
   requesting it as a Float. No JavaScript provider can satisfy that scenario — not through a defect
   but because the distinction does not exist in the language. Every JS suite therefore leaves the
   capability undeclared and the scenario is reported as skipped, which is locally honest but reads
   as "JS providers have a known bug" when they do not. Options: accept per-language undeclaration
   as-is, mark the capability explicitly language-scoped, or split it into "does not truncate a
   float" (universal) and "keeps the two numeric types distinct" (only where the language has two).
   Surfaced by js-sdk-contrib#1606.

8. Who owns conformance claims? Split out into
   #424 — a machine-readable report format, how
   to collect reports from providers hosted outside this org, and how to present them without
   creating an incentive to declare fewer capabilities.

Per-language tracking

• Java — proof of concept complete: java-sdk-contrib#1829
• JavaScript / TypeScript — proof of concept in review: js-sdk-contrib#1607
• Go — proof of concept in review: go-sdk-contrib#938
• .NET
• Python — proof of concept in review: python-sdk-contrib#410
• PHP
• Ruby
• Kotlin / Swift (client-side)

Linked work

Where	What
spec#423	this repo — Appendix F, plus the three artifacts under specification/assets/provider-tck/
java-sdk-contrib#1829	Java implementation issue
java-sdk-contrib#1830	tools/provider-tck — the suite, step definitions, containerised base class, flagd adoption
java-sdk-contrib#1837	in-process control path, so backend-less providers can run the suite without Docker
java-sdk-contrib#1838	switches the Java module to consume this repo's artifacts via submodule (depends on #423)
go-sdk-contrib#938	Go implementation issue
go-sdk-contrib#940	tools/provider-tck — the suite, godog step definitions, in-process control, three self-tests
go-sdk-contrib#941	flagd adoption for both resolvers, plus the control-API client
python-sdk-contrib#410	Python implementation issue
python-sdk-contrib#409	tools/openfeature-provider-tck — the pytest-bdd implementation, shipped as a pytest plugin
js-sdk-contrib#1607	JavaScript implementation issue
js-sdk-contrib#1606	libs/shared/provider-tck — the jest-cucumber implementation
spec#424	publishing conformance reports — follow-up design discussion

spec#423 takes a position on two of the open questions above and leaves the rest open:

• Q1, directory layout — proposes specification/assets/provider-tck/{gherkin,flags,openapi}/, alongside the existing assets/gherkin/. A subdirectory is forced rather than chosen: assets/gherkin/evaluation.feature already exists and means something different (it tests an SDK, not a provider).
• Q4, @strict-numeric-typing — keeps it as a capability for now, but the appendix states in prose that it is not an optional feature and that not declaring it is an admission of a known bug. The "known deviations" idea is recorded as still open.

Q2 (versioning), Q3 (flag-set format), Q5 (context passthrough), Q6 (reason values) and Q7 (conformance claims) are untouched by #423 and remain open for discussion here.

Findings from the language implementations

Bugs the suite surfaced that were not previously tracked. Listed here because the point of the
exercise is whether it finds real things.

Finding	Where	Tracked
flagd narrows a float flag to an integer, returning 0 with no error code	Java provider, both resolvers	java-sdk-contrib#1830
MultiProvider never subscribes to its children, swallowing their events	java-sdk	java-sdk#1882
The in-memory provider cannot update its flag set or emit PROVIDER_CONFIGURATION_CHANGED, which Appendix A requires	go-sdk	go-sdk#530
The flagd RPC resolver never emits PROVIDER_STALE, while its in-process resolver does	Go provider	go-sdk-contrib#939
A boolean satisfies an Integer request, because bool subclasses int	python-sdk	python-sdk#619
The in-memory provider cannot update its flag set (Appendix A)	python-sdk	python-sdk#620
The in-memory provider does update and emit correctly — the reference behaviour the other two should grow	js-sdk	n/a (positive result)

Two of these are worth dwelling on. The flagd resolvers report an outage differently from each
other, which is precisely the vendor-neutrality claim the suite is meant to check. And the Python
boolean-as-integer bug could not have been found by any other language's suite — bool is a
subclass of int only in Python — which is the clearest argument yet for implementing this in more
than one language rather than treating one implementation as sufficient.

The in-memory provider gap has now been found independently in two SDKs, which suggests it is worth
raising as a cross-SDK item rather than two separate bugs.

Prior art in this ecosystem

• open-feature/test-harness — the flagd-oriented
  Gherkin suite and testbed this work generalises.
• open-feature/flagd-testbed — the launchpad
  control API the standardised one is derived from.
• dev.openfeature.contrib.tools:flagd-api-testkit in java-sdk-contrib — an existing "abstract
  suite + SPI factory, features packaged in the JAR" testkit whose adoption ergonomics the Java PoC
  copies.