Proposal: Provider TCK — language-agnostic conformance tests for OpenFeature providers

[aepfli]

Problem

There is currently no standard way to verify that an OpenFeature provider correctly implements the specification. Each provider is tested in isolation, with test strategies that differ across languages and vendors. This makes it hard to trust that replacing one provider with another will produce identical behavior — which is the core promise of OpenFeature.

The closest thing we have is the flagd TCK, which validates flagd-specific behavior. It proves the model works, but it is not general-purpose and cannot be adopted by other providers without significant rework.

Proposal

A Provider TCK (Technology Compatibility Kit): a suite of language-agnostic conformance tests written in Gherkin that any provider can run against by wiring up a thin harness shim. The test cases are derived directly from the OpenFeature specification — they test the provider contract, not backend logic.

What gets tested

The TCK covers only what the spec requires of providers:

• Typed resolution — correct value and variant returned for all flag types (boolean, string, integer, float, object); type mismatch returns TYPE_MISMATCH, not a coercion
• Error codes — FLAG_NOT_FOUND, TYPE_MISMATCH, PARSE_ERROR, GENERAL surfaced correctly; default value returned on error; no thrown exceptions from resolution
• Reason codes — STATIC, TARGETING_MATCH, DEFAULT, UNKNOWN, CACHED, ERROR returned correctly
• Provider lifecycle — READY, ERROR, STALE states reached at the right times; correct events emitted (PROVIDER_READY, PROVIDER_ERROR, PROVIDER_STALE, PROVIDER_CONFIGURATION_CHANGED)
• Flag change handling — PROVIDER_CONFIGURATION_CHANGED fires when backend signals a change; re-evaluation reflects the updated flag
• Context passthrough — evaluation context (merged: global + transaction + invocation) reaches the backend correctly
• Shutdown — shutdown() cleans up without errors

The question is also what is a must, and what a should. eg. STALE is an optional state. We should be able to also activate and deactivate this easily

What is explicitly not tested: backend evaluation logic, targeting rule behavior, flag modification APIs, or how the provider communicates with its backend.

Architectural-Idea (just one possible idea)

Gherkin scenario
      │
      ▼
TCK runner (per-language, thin)
      │                    │
      ▼                    ▼
Provider under test    Mock backend (harness-controlled)

The harness controls the mock backend — not the provider. This is what makes disconnect testing, flag change events, and error conditions deterministic and reproducible across providers.

Each provider author implements one interface (approximately) or a defined docker image with rest endpoints.

interface ProviderHarness {
    void setFlag(String key, FlagDefinition flag);
    void removeFlag(String key);
    void triggerChange();
    void disconnect();
    void reconnect();
}

That's the full surface area. Point your provider at localhost:{port}, implement the harness, run the suite.

Language agnosticism

Test cases are defined once in Gherkin. Each language ecosystem provides:

1. A step definition library that maps Gherkin steps to SDK calls
2. A mock backend appropriate for that language/protocol

The mock backend speaks whatever protocol the provider expects (REST, gRPC, etc.). The harness drives it via the control interface above. Two providers using different protocols can run identical Gherkin scenarios.

Relation to the flagd TCK

The flagd TCK is a direct input to this work. The generalized TCK is built fresh so the contract is clean, but flagd should require minimal effort to adopt it — the scenarios are largely the same, and the harness maps cleanly to flagd's existing test infrastructure. That migration is part of validating the design.

PoC

A PoC will accompany this proposal to validate feasibility. It will:

• Define the harness interface
• Implement a simple in-process mock backend
• Run a representative subset of scenarios against an existing Java provider
• Show what it takes to wire up a second provider (to prove the shim is thin)

Open questions

• Where does this live? A new openfeature/provider-tck repo, or within openfeature/spec?
• Should the mock backend be a shared out-of-process sidecar (one per protocol), or in-process per language?

[toddbaert]

I think this is a great idea. I have a feeling this should be a spec appendix, since we already have similar gherkin there. I suspect we need more than just a gherkin suite... which makes me thing maybe a new repo for that actual code?

Not quite sure how to split it up, but I think that's an implementation detail.

@aepfli how much code besides gherkin do you foresee needing to exist, long term? I don't want to add too much actual code to the spec repo if we can avoid it.

[aepfli]

I would actually try to do something like the flagd-api-testkit in the contrib repos. For each language a specific gherkin suite with already in place steps.

Maybe we define a docker image "format".

The docker image is the backend, or mocks it. Etc.

But we have rest endpoints like for the flagd harness, to switch flags, to stop the connection, to start it, and maybe something like a restart with time.

The last piece would be a setup file within each provider bootstrapping the provider and the test based on the testing artifact.

The testing artifact should already bundle everything, like the gherkin files and the gherkin steps.

This way, the implementation on the provider side is minimal (a backend(-mock) with rest controls reusable for all languages), and for each language the instrumentalization of the provider for the tests.

I need to run experiments, but it is working separately in this kind of way for flagd already.

Wdyt?