feat(provider-tck): add the Python conformance suite for OpenFeature providers

[aepfli]

Closes #410
Part of open-feature/spec#417 — the cross-language tracking issue for the provider conformance suite. Java is the reference (java-sdk-contrib#1830); Go is in review (go-sdk-contrib#940).

This one was built and run locally, unlike the Go implementation: 56 passed, 7 skipped, 2 xfailed, with ruff and mypy --strict clean. Every design decision below was probed against pytest-bdd 8.1 rather than assumed.

What this is

A conformance suite any Python provider can adopt to verify it implements the provider contract — the Python implementation of Appendix F, running the same Gherkin scenarios, against the same canonical flag set, driven through the same control API as every other language's TCK. That shared basis is the point: "conformant" only means something if the question is identical everywhere.

It uses pytest-bdd, the same runner openfeature-provider-flagd and openfeature-flagd-api-testkit already depend on, so an adopting package gains no new test framework.

The adoption surface

One fixture and one call:

import pytest
from pytest_bdd import scenarios

from openfeature.contrib.tools.provider_tck import Capability, TckConfig, features_path


@pytest.fixture(scope="session")
def tck_config():
    control = MyBackendControl()
    return TckConfig(
        name="my-provider",
        control=control,
        new_provider=lambda: MyProvider(control.address),
        capabilities={Capability.EVENTS, Capability.OBJECT},
    )


scenarios(features_path())

No conftest.py, and nothing to import for the steps. The vocabulary ships as a pytest plugin registered through a pytest11 entry point. This is the nicest of the three implementations so far, and it is not an accident — pytest-bdd resolves steps through the fixture system, and fixtures from an installed plugin are visible to every test.

The feature files and canonical flag set are packaged with the distribution, so adopting needs no git submodule.

Four things I probed rather than assumed

Having a working toolchain this time, each of these was verified against pytest-bdd 8.1 before the design depended on it:

Question	Answer
Does pytest-bdd turn Gherkin tags into markers, including dashed ones like @configuration-change?	yes — getattr(pytest.mark, tag) handles them
Does pytest.skip() from an autouse fixture report skipped with the reason?	yes, natively — no reporting machinery needed, unlike Go
Does scenarios() accept an absolute path into an installed package?	yes
Do step definitions work from a plugin rather than the test module?	yes — this is what removes the conftest.py

Two things that only showed up by running it, both now fixed and commented:

• pytest-bdd creates markers without registering them, so every tag raised PytestUnknownMarkWarning — noise at best, a hard failure under -W error. The plugin registers them in pytest_configure.
• The capability gate silently did nothing when guarded on request.fixturenames. pytest-bdd resolves a step's fixtures lazily as each step runs, so tck_config is not in fixturenames at setup time, and @unavailable scenarios ran against a config that never declared it. The gate now keys off the node's markers, which are on the item itself.

That second one is exactly the failure mode the suite exists to prevent — a gate that looks right and quietly passes everything — so it is pinned by a test.

Capabilities

A scenario whose capability was not declared is reported as skipped, with the reason — never as passed:

SKIPPED provider does not declare capability @stale.
        Declared: @events @object @strict-numeric-typing

Self-tests

Suite	Subject	Why
test_in_memory_conformance	the SDK's InMemoryProvider	reference adoption for a backend-less provider, and the Docker-free canary
test_controllable_conformance	ControllableInMemoryProvider	the only suite that exercises the configuration-change path — see finding 2
test_in_process_control	InProcessControl	pins what the Gherkin cannot assert about itself

There is no multi-provider suite because Python has no multi-provider — worth noting as its own gap.

Findings

1. A boolean satisfies an Integer request — and this is Python-only

boolean-flag evaluated through get_integer_details returns True, reason STATIC, no error code. The specification requires the code default and TYPE_MISMATCH.

type_map = {FlagType.INTEGER: int, ...}
if not isinstance(value, py_type):
    return TypeMismatchError(...)

bool is a subclass of int in Python, so isinstance(True, int) is True and the check passes. The application gets a value that behaves as 1, with nothing to indicate anything went wrong.

The identical scenario passes in Java and Go. No suite in another language could ever have caught this — which is a fair advertisement for the "multiple implementations" argument, and arrived on the suite's first real run.

Tracked as open-feature/python-sdk#619. The self-test marks that one row xfail(strict=True) with a pointer to the issue, so it stays visible in the report and fails the moment it starts passing, which forces the marker's removal when the SDK is fixed.

2. The in-memory provider cannot update its flag set

Appendix A requires it, and Python's copies its mapping in the constructor and exposes nothing to change it. Same class of gap as go-sdk#530, found independently in a second SDK.

Only half the machinery is missing — AbstractProvider already supplies emit_provider_configuration_changed — so ControllableInMemoryProvider here is a small subclass, not a reimplementation: every resolution decision is still the SDK's. It should port back as a method. Tracked as open-feature/python-sdk#620.

Verification

Check	Result
pytest tests	56 passed, 7 skipped, 2 xfailed
ruff check (repo config)	clean
mypy --strict	clean, 13 source files
Assets byte-identical to spec#423	yes

Run in a clean venv against openfeature-sdk 0.8.4.

Known gaps

• The assets are vendored, not submoduled. A follow-up will source them from the spec repo at build time, as openfeature-flagd-api-testkit already does for the flagd harness.
• No HTTP control client yet — it arrives with the first containerised adopter (flagd or OFREP).
• Evaluation context passthrough is unverifiable without an echo endpoint on the control API.
• Caching, hooks and flag metadata are not covered.

Open questions

1. Is tools/openfeature-provider-tck the right home, alongside the flagd testkit?
2. The xfail(strict=True) for a known SDK deviation is a local answer to spec#417's open question 4 ("is a known-deviations concept needed?"). Does that shape look right before it becomes a pattern?
3. Should ControllableInMemoryProvider live here at all, or should the SDK fix land first and this package depend on it?

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 78.41727% with 120 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.03%. Comparing base (92c5f49) to head (d6de5dc).

Files with missing lines	Patch %	Lines
...ure/contrib/tools/provider_tck/steps/flag_steps.py	72.97%	30 Missing ⚠️
...re/contrib/tools/provider_tck/steps/event_steps.py	61.19%	26 Missing ⚠️
...contrib/tools/provider_tck/steps/provider_steps.py	53.33%	21 Missing ⚠️
...c/openfeature/contrib/tools/provider_tck/values.py	76.78%	13 Missing ⚠️
...rc/openfeature/contrib/tools/provider_tck/state.py	84.93%	11 Missing ⚠️
...c/openfeature/contrib/tools/provider_tck/config.py	83.63%	9 Missing ⚠️
...openfeature/contrib/tools/provider_tck/__init__.py	76.47%	4 Missing ⚠️
...c/openfeature/contrib/tools/provider_tck/plugin.py	94.11%	2 Missing ⚠️
...enfeature/contrib/tools/provider_tck/capability.py	96.87%	1 Missing ⚠️
.../openfeature/contrib/tools/provider_tck/control.py	93.75%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #409      +/-   ##
==========================================
- Coverage   95.64%   92.03%   -3.62%     
==========================================
  Files          24       59      +35     
  Lines        1057     2334    +1277     
==========================================
+ Hits         1011     2148    +1137     
- Misses         46      186     +140     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.