feat(provider-tck): emit a machine-readable conformance report

[aepfli]

Stacked on #409 — base is feat/provider-tck, so the diff here is only the report emitter. Part of open-feature/spec#424; the schema is open-feature/spec#425. This mirrors the Go implementation in go-sdk-contrib.

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json.

$ PROVIDER_TCK_REPORT_DIR=./reports uv run pytest tools/openfeature-provider-tck/tests
provider-tck [in-memory]: report written to reports/in-memory.json (1 failed, 5 not-declared, 23 passed)
provider-tck [controllable-in-memory]: report written to reports/controllable-in-memory.json (1 failed, 4 not-declared, 24 passed)

47 passed, 9 skipped, 2 xfailed in 5.41s

Both documents account for all 29 scenarios. A third suite, generated by the tests, produces all four outcomes at once: {"passed": 22, "not-declared": 5, "not-applicable": 1, "failed": 1}, again 29 of 29.

Why an environment variable

It is not a TckConfig field, because emitting a report is a property of the run and not of the code. CI sets it, a developer running the suite locally does not, and no adopter changes a line to publish one. Unset means no report, which is the default and is not an error. Several suites in one pytest session each write their own file, so flagd's RPC and in-process resolvers would not collide — and suite names are chosen to read well in a failure message rather than to be path-safe, so flagd/rpc is sanitised to flagd-rpc.json instead of writing outside the directory it was given.

Why every scenario is listed

Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed. Nothing downstream can check that against a summary line, so the report records the outcome of every scenario individually — and is required to be complete, because a document that quietly dropped what it skipped would satisfy the letter of the rule and still mislead whoever read it.

pytest, unlike godog, already reports a skip honestly, so the divergence between the runner and the document shows up somewhere else here. The one scenario the Python SDK cannot satisfy is marked xfail(strict=True) (see finding 1 in the package README), so the run exits zero. The provider still did not satisfy it, and the document says failed with the reason. An expected failure is a recorded deviation, not an excused one.

Four outcomes rather than two:

Outcome	Means
passed	ran and passed
failed	ran and failed, including a known deviation marked xfail
not-declared	skipped because the provider did not declare a capability the scenario is tagged with
not-applicable	skipped for any other reason — a marker an adopter applied, a step calling pytest.skip

The not-declared / not-applicable split is decided from the scenario's tags and the suite's declared capabilities, never from the wording of the skip message: the message is prose and the distinction is not.

Scenarios are enumerated at collection and resolved into records only at the end of the session. That ordering is load-bearing. A scenario skipped by a marker never runs a fixture, so an emitter that learned of a scenario when its fixtures ran would leave it out of the document entirely — which is the exact failure mode this format exists to rule out. An autouse fixture, requested by the capability gate so it is guaranteed to run first, is only what tells the emitter which TckConfig a module is testing.

Why the provider's own metadata name

provider.name is what the provider reports through get_metadata(), not TckConfig.name. TckConfig.name is chosen to read well in a failure message — flagd-rpc — which makes it the configuration, and it is reported as provider.configuration. One provider with two materially different modes therefore produces two reports that are not interchangeable, which is the property the schema's configuration field exists for. When a suite skipped every scenario and so never saw a provider, the name falls back to the suite name rather than the empty string the schema would reject.

backend.controlApi is read off an optional control_api property on the BackendControl, not a new protocol member: adding one would make every existing control incomplete for the sake of one string, and a control that stays quiet simply omits the field. InProcessControl reports in-process; the plain in-memory suite's control says nothing, and its report has no controlApi — which is the intended behaviour, visible in the two files above.

Identity, captured at build time

tck.specRevision and tck.assetsTree come from a generated spec_revision.json that hatch_build_sync.py writes beside the copied assets and hatch_build.py force-includes into the wheel and sdist. It has to be captured at build time: the submodule that knows the answer is not in the distribution, so an installed copy has nothing left to ask.

The tree hash is carried as well as the commit because it identifies the assets alone — unchanged by unrelated edits elsewhere in the specification, so two runs of identical assets agree even when pinned to different commits, and checkable rather than asserted.

A build that cannot reach git warns and records "unknown" (seven characters, the schema's minimum) rather than inventing a commit. The build never hard-fails on it. subprocess.run with a list argv and pathlib throughout, so the step works on Windows, Linux and macOS.

Verification

Run on WSL Ubuntu, Python 3.10 (the repo's pinned version), against the schema as it stands on feat/provider-tck-report-schema.

Package test suite. 78 passed, 9 skipped, 2 xfailed. test_report.py is new and adds 24 of those; four of its cases run a generated one-fixture adoption in a subprocess, because the properties it checks are properties of a whole pytest session.

Lint and types. ruff check, ruff format --check and mypy (whose scope includes tests) are all clean. No dependencies were added, so uv.lock is untouched and uv sync --frozen is unaffected.

Schema validation. Every emitted report validated with jsonschema as Draft 2020-12 against specification/assets/provider-tck/report/conformance-report.schema.json:

Report	Result	Outcomes	Total
in-memory.json	VALID	{"passed": 23, "not-declared": 5, "failed": 1}	29
controllable-in-memory.json	VALID	{"passed": 24, "not-declared": 4, "failed": 1}	29
report-fixture.json (generated by the tests)	VALID	{"passed": 22, "not-declared": 5, "not-applicable": 1, "failed": 1}	29

Totals add up. In each report the outcome counts sum to the number of scenario entries (29), and the 29 (feature, name) pairs are distinct. test_every_collected_scenario_appears_exactly_once checks this against pytest's own --collect-only count rather than a number written down beside it, so adding a scenario to the specification cannot leave it passing while the report loses one.

Tree hash is checkable. git rev-parse dfa16586d91ca020ef1b3b82a7c972d833ff8f29:specification/assets/provider-tck in the submodule returns 904aa7d5fd7a856a4f92ace24355bd1987143abc, which is what the reports carry.

Packaging. uv build produces a wheel containing spec_revision.json alongside the four feature files, the canonical flag set and control-api.yaml. Building an sdist, unpacking it, deleting .git and building a wheel from that — no submodule, no git — succeeds and carries the revision forward unchanged. The degraded path was exercised directly by pointing the sync at a directory that is not a repository: it warns twice and writes {"specRevision": "unknown", "assetsTree": ""}, which still validates.

Failure to write. Verified end to end by pointing PROVIDER_TCK_REPORT_DIR at a path under a regular file: the message appears and the session exits non-zero on an otherwise-passing run. A run that asked for a report and silently did not get one is how a publishing pipeline ends up serving a stale result forever.

Things worth raising, rather than glossing

A report cannot name a Scenario Outline row portably. Every row of an outline shares one scenario name, and the schema has nowhere to put the row, so a report keyed on (feature, name) would carry several indistinguishable entries — including, here, one row of Requesting the wrong type returns the code default that differs in outcome from its ten siblings. This implementation qualifies the name with pytest's example id (... [boolean-flag-Integer-1]), which is unambiguous but is not what godog would produce for the same row, so Go and Python reports do not key identically on outline rows. Either the schema needs a discriminator or the format needs to say how the row is spelled. Raised for open-feature/spec#424.

There is no overall verdict, and a mandatory failure surfaces nowhere but scenarios. The scenario the Python SDK fails carries no capability tag — it is a mandatory scenario — so it does not roll up into capabilities, and the top level has no field that says the run failed. in-memory.json shows every declared capability as passed while one mandatory scenario failed. A consumer that reads only capabilities would conclude the provider is fine. Worth deciding whether the format wants a top-level verdict or whether consumers are expected to aggregate scenarios themselves.

knownDeviation exists only on capabilityResult, not on scenarioResult. The distinction it draws — "chose not to implement an optional feature" versus "has a known bug" — is exactly the one an xfailed scenario is recording, and there is nowhere to put it. The reason string carries the issue reference as prose instead.

reason is not required for a non-passed outcome. The schema's own description says it is "required in practice"; nothing enforces it. This emitter always supplies one, but a report that omits it would validate.

Not done here: CI does not upload the reports as artifacts. That belongs with whatever consumes them, and can follow.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}