Add `tools/openfeature-provider-tck`: a Python conformance suite for OpenFeature providers

[aepfli]

Part of open-feature/spec#417, the cross-language tracking issue for the provider conformance suite. This issue is the Python half; the language-agnostic artifacts (Gherkin, canonical flag set, control API) are proposed for the spec repo in spec#423. Sibling issues: java-sdk-contrib#1829, go-sdk-contrib#938.

Summary

A new package, tools/openfeature-provider-tck, containing the Python implementation of the OpenFeature Provider TCK: the canonical Gherkin, pytest-bdd step definitions, and the lifecycle that owns provider registration, event awaiting and per-scenario backend reset.

It uses pytest-bdd, the runner openfeature-provider-flagd and openfeature-flagd-api-testkit already depend on, so an adopting package gains no new test framework.

Status: proof of concept, verified locally — 56 passed, 7 skipped, 2 xfailed, with ruff and mypy --strict clean.

The adoption surface

One fixture and one call. The step definitions ship as a pytest plugin registered through a pytest11 entry point, so there is no conftest.py to write and nothing to import for the vocabulary:

@pytest.fixture(scope="session")
def tck_config():
    control = InProcessControl()
    return TckConfig(
        name="my-provider",
        control=control,
        new_provider=control.new_provider,
        capabilities={Capability.EVENTS, Capability.OBJECT},
    )


scenarios(features_path())

Package layout

tools/openfeature-provider-tck/
├── pyproject.toml                    pytest11 entry point
├── README.md
└── src/openfeature/contrib/tools/provider_tck/
    ├── features/*.feature            canonical Gherkin, packaged
    ├── flag_data/canonical-flags.json
    ├── control-api.yaml
    ├── capability.py                 capability ↔ Gherkin tag
    ├── config.py                     TckConfig — the adoption surface
    ├── control.py                    BackendControl / ConnectionControl
    ├── inprocess.py                  in-process control for backend-less providers
    ├── provider.py                   ControllableInMemoryProvider
    ├── plugin.py                     markers, capability gate, fixtures
    ├── state.py
    └── steps/                        the shared vocabulary

Structure follows tools/openfeature-flagd-api-testkit, which already solved the "ship Gherkin with the package" problem via importlib.resources.

Design points worth challenging

1. Capability gating uses pytest.skip from an autouse fixture, so an undeclared capability is reported as skipped with the reason — which the specification requires. The gate keys off request.node.iter_markers() rather than request.fixturenames, because pytest-bdd resolves a step's fixtures lazily and guarding on fixturenames silently disabled the gate entirely.
2. Markers are registered in pytest_configure, because pytest-bdd creates tag markers with getattr(pytest.mark, tag) without registering them — every tag otherwise raises PytestUnknownMarkWarning, which is fatal under -W error.
3. A known SDK deviation is marked xfail(strict=True) rather than hidden, so it stays visible and fails the moment it starts passing. This is a local answer to spec#417's open question 4 about a "known deviations" concept.

Findings

Both confirmed by running the suite, both filed separately:

• python-sdk#619 — a boolean satisfies an Integer request, because bool subclasses int and the client type-checks with isinstance. Python-specific: the identical scenario passes in every other language.
• python-sdk#620 — InMemoryProvider cannot update its flag set, which Appendix A requires. Same gap independently found in go-sdk.

Scope

• tools/openfeature-provider-tck — the suite, step definitions, in-process control, self-tests (#409)
• flagd adoption for both resolvers, plus the HTTP control-API client
• source the assets from the open-feature/spec submodule instead of vendoring them
• OFREP adoption