feat(provider-tck): emit a machine-readable conformance report

tools/openfeature-provider-tck/.gitignore

@@ -5,3 +5,6 @@
 src/openfeature/contrib/tools/provider_tck/features/
 src/openfeature/contrib/tools/provider_tck/flag_data/
 src/openfeature/contrib/tools/provider_tck/control-api.yaml
+# Generated alongside them, from the submodule pin, so a conformance report can
+# name the spec revision it ran against.
+src/openfeature/contrib/tools/provider_tck/spec_revision.json

tools/openfeature-provider-tck/README.md

@@ -204,19 +204,88 @@ is recorded by the pin and nowhere else, so the two cannot drift apart unnoticed
 
 This mirrors what `openfeature-flagd-api-testkit` already does for the flagd test harness.
 
+## Conformance reports
+
+Set `PROVIDER_TCK_REPORT_DIR` and each suite writes a machine-readable record of its run to
+`<dir>/<name>.json`, conforming to the [report schema][report-schema] in the specification.
+
+```console
+$ PROVIDER_TCK_REPORT_DIR=./reports pytest
+provider-tck [in-memory]: report written to reports/in-memory.json (1 failed, 5 not-declared, 23 passed)
+
+$ jq '.scenarios | group_by(.outcome) | map({(.[0].outcome): length}) | add' reports/in-memory.json
+{
+  "failed": 1,
+  "not-declared": 5,
+  "passed": 23
+}
+```
+
+It is an environment variable rather than a `TckConfig` field so that emitting a report is a property
+of the *run* and not of the code: CI sets it, a developer running the suite locally does not, and no
+adopter changes a line to publish one. Unset means no report, which is not an error. Several suites
+in one pytest session each write their own file, so flagd's two resolvers would not collide.
+
+### Why every scenario is listed
+
+Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped
+**with the reason** and never as passed. A consumer cannot check that against a summary line, so the
+report records the outcome of *every* scenario individually — and is required to be complete, because
+a document that quietly dropped what it skipped would satisfy the letter of the rule and still
+mislead whoever read it.
+
+Which also means the report is not a transcription of pytest's summary. The run above finishes green:
+the one scenario the Python SDK cannot satisfy is marked `xfail` (finding 1), so pytest counts it as
+expected and exits zero. The provider still did not satisfy it, and the document says `failed` with
+the reason — an expected failure is a recorded deviation, not an excused one.
+
+Four outcomes rather than two, because "did not run" is not one thing:
+
+| Outcome | Means |
+| --- | --- |
+| `passed` | the scenario ran and passed |
+| `failed` | the scenario ran and failed, including a known deviation marked `xfail` |
+| `not-declared` | skipped because the provider did not declare a capability the scenario is tagged with |
+| `not-applicable` | skipped for any other reason — a marker an adopter applied, a step calling `pytest.skip` |
+
+### What identifies a report
+
+`tck.specRevision` and `tck.assetsTree` come from `spec_revision.json`, which `hatch_build_sync.py`
+generates from the submodule alongside the copied assets. It has to be captured at build time: the
+submodule is not in the wheel, so an installed copy has nothing left to ask. A build that cannot
+reach git — an unpacked sdist, say — warns and records `unknown` rather than inventing a commit.
+
+The tree hash is carried as well as the commit because it identifies the assets alone. It is
+unchanged by unrelated edits elsewhere in the specification, so two runs that executed identical
+assets report the same value even when pinned to different commits — and it is checkable, since
+`git rev-parse <specRevision>:specification/assets/provider-tck` must reproduce it.
+
+`provider.name` is what the provider reports through its own metadata, not `TckConfig.name`.
+`TckConfig.name` is chosen to read well in a failure message — `flagd-rpc` — which makes it the
+*configuration*, and it is reported as such. One provider with two materially different modes
+produces two reports that are not interchangeable.
+
+`backend.controlApi` is read off an optional `control_api` property on your `BackendControl`,
+returning `"http"` or `"in-process"`. It is not a member of the protocol: adding one would make every
+existing control incomplete for the sake of one string, and a control that stays quiet simply omits
+the field.
+
 ## The self-tests
 
 | Suite | Subject | Why |
 | --- | --- | --- |
 | `test_in_memory_conformance` | the SDK's `InMemoryProvider` | reference adoption for a backend-less provider |
 | `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 |
 | `test_in_process_control` | `InProcessControl` | pins what the Gherkin cannot assert about itself |
+| `test_report` | the conformance report | checks the two properties a consumer is entitled to assume |
 
 ```
-54 passed, 9 skipped, 2 xfailed
+78 passed, 9 skipped, 2 xfailed
 ```
 
-No Docker, no network, under a second.
+No Docker and no network. The conformance suites take under a second; `test_report` takes most of a
+minute, because the properties it checks are properties of a whole pytest session and it runs four of
+them in subprocesses to check them.
 
 Neither in-memory suite declares `@lifecycle`, so the three lifecycle scenarios are skipped in both.
 That is the point: with no backend to reach, they would pass without testing anything — which is
@@ -228,7 +297,14 @@ what they did while the feature was gated on `@events`.
   cannot assert one *reached* the backend. That needs an echo operation on the control API.
 - **No HTTP control client yet.** It arrives with the first containerised adopter.
 - **Caching, hooks and flag metadata** are not covered.
-
+- **A report cannot name a Scenario Outline row portably.** Every row of an outline shares one
+  scenario name, and the report schema has nowhere to put the row, so several entries would be
+  indistinguishable — including, here, one that differs in outcome from its siblings. This
+  implementation qualifies the name with pytest's example id (`... [boolean-flag-Integer-1]`), which
+  is unambiguous but is not what another language would produce for the same row. Raised on
+  [open-feature/spec#424](https://github.com/open-feature/spec/issues/424).
+
+[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json
 [appendix-a]: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md
 [appendix-f]: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md
 [spec]: https://github.com/open-feature/spec

tools/openfeature-provider-tck/hatch_build.py

@@ -18,15 +18,28 @@
 # the single definition of what gets copied where -- would not be importable.
 sys.path.insert(0, str(Path(__file__).parent))
 
-from hatch_build_sync import FILES, PACKAGE_REL, SPEC_ASSETS, TREES, sync
+from hatch_build_sync import (
+    FILES,
+    PACKAGE_REL,
+    REVISION_FILE,
+    SPEC_ASSETS,
+    TREES,
+    sync,
+)
 
 
 class SpecAssetsCopyHook(BuildHookInterface):
     PLUGIN_NAME = "spec-assets-copy"
 
     def initialize(self, version: str, build_data: dict) -> None:
         root = Path(self.root)
-        copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES]
+        # The generated revision file travels with the assets it describes. It
+        # has to be built here rather than read at run time, because the
+        # submodule that knows the answer is not in the wheel and a conformance
+        # report has to name the revision it ran against.
+        copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES] + [
+            root / PACKAGE_REL / REVISION_FILE
+        ]
 
         # Building from a checkout: refresh from the submodule, so what ships is
         # always the revision the pin names. Building from an sdist: there is no

tools/openfeature-provider-tck/hatch_build_sync.py

@@ -10,11 +10,16 @@
 needs no submodule: the copies are inside the distribution.
 """
 
+import json
 import shutil
+import subprocess
+import warnings
 from pathlib import Path
 
 ROOT = Path(__file__).parent
-SPEC_ASSETS = (ROOT / "spec/specification/assets/provider-tck").resolve()
+SPEC_ROOT = (ROOT / "spec").resolve()
+ASSETS_PATH_IN_SPEC = "specification/assets/provider-tck"
+SPEC_ASSETS = (SPEC_ROOT / ASSETS_PATH_IN_SPEC).resolve()
 PACKAGE_REL = Path("src/openfeature/contrib/tools/provider_tck")
 DEST_BASE = ROOT / PACKAGE_REL
 
@@ -28,6 +33,27 @@
 TREES = [("gherkin", "features"), ("flags", "flag_data")]
 FILES = [("openapi/control-api.yaml", "control-api.yaml")]
 
+REVISION_FILE = "spec_revision.json"
+"""Which revision of the specification the copied assets came from.
+
+Recorded at build time because the answer is only available at build time: the
+submodule that holds it is not in the wheel, and a conformance report that cannot
+name the revision it ran against cannot be compared with another. It is generated
+by the same command that copies the assets, which is what keeps the two from
+disagreeing.
+
+Not committed, for the same reason the assets are not: the submodule pin is the
+single record of which revision this package targets.
+"""
+
+UNKNOWN_REVISION = "unknown"
+"""Seven characters, the minimum the report schema accepts.
+
+A build that cannot reach git says it does not know rather than inventing a
+commit, and still produces a document that validates. Which happens for real:
+building from a source tarball has no ``.git`` to ask.
+"""
+
 
 def sync() -> None:
     if not SPEC_ASSETS.exists():
@@ -51,6 +77,50 @@ def sync() -> None:
             dest.unlink()
         shutil.copy2(SPEC_ASSETS / src_name, dest)
 
+    write_revision()
+
+
+def write_revision() -> None:
+    """Record the spec commit and the asset tree these copies came from.
+
+    The tree hash is carried as well as the commit because it identifies the
+    assets alone: it does not change when an unrelated part of the specification
+    does, so two runs that executed identical assets report the same value even
+    when pinned to different commits. It is also checkable rather than merely
+    asserted, since ``git rev-parse <commit>:specification/assets/provider-tck``
+    must reproduce it.
+    """
+    commit = _git("rev-parse", "HEAD") or UNKNOWN_REVISION
+    tree = _git("rev-parse", f"HEAD:{ASSETS_PATH_IN_SPEC}") or ""
+    (DEST_BASE / REVISION_FILE).write_text(
+        json.dumps({"specRevision": commit, "assetsTree": tree}, indent=2) + "\n",
+        encoding="utf-8",
+    )
+
+
+def _git(*args: str) -> str:
+    """Run git inside the submodule, returning its output or an empty string.
+
+    A build must not hard-fail because git is absent or the checkout is not a
+    repository -- both are ordinary when building from an unpacked sdist. The
+    failure is reported as a warning and the identity degrades to ``unknown``,
+    which is legible in the resulting report rather than silently wrong.
+    """
+    command = ["git", "-C", str(SPEC_ROOT), *args]
+    try:
+        completed = subprocess.run(  # noqa: S603
+            command, capture_output=True, check=True, text=True
+        )
+    except (OSError, subprocess.CalledProcessError) as error:
+        warnings.warn(
+            f"could not determine the spec revision ({' '.join(command)}: {error}); "
+            f"conformance reports from this build will not name the revision they "
+            f"ran against",
+            stacklevel=2,
+        )
+        return ""
+    return completed.stdout.strip()
+
 
 if __name__ == "__main__":
     sync()

tools/openfeature-provider-tck/pyproject.toml

@@ -58,6 +58,10 @@ artifacts = [
     "src/openfeature/contrib/tools/provider_tck/features/",
     "src/openfeature/contrib/tools/provider_tck/flag_data/",
     "src/openfeature/contrib/tools/provider_tck/control-api.yaml",
+    # Which spec revision those assets came from, generated beside them. The
+    # submodule is not in the wheel, so a conformance report emitted by an
+    # installed copy has no other way to name the revision it ran against.
+    "src/openfeature/contrib/tools/provider_tck/spec_revision.json",
 ]
 
 [tool.hatch.build.hooks.custom]

tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py

@@ -62,15 +62,19 @@ def tck_config():
     ControllableInMemoryProvider,
     canonical_flag_set,
 )
+from .report import REPORT_DIR_ENV, SCHEMA_VERSION, Outcome
 
 __all__ = [
     "ALL_CAPABILITIES",
     "CHANGING_FLAG_KEY",
+    "REPORT_DIR_ENV",
+    "SCHEMA_VERSION",
     "BackendControl",
     "Capability",
     "ConnectionControl",
     "ControllableInMemoryProvider",
     "InProcessControl",
+    "Outcome",
     "TckConfig",
     "UnsupportedControlError",
     "canonical_flag_set",

tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py

@@ -101,6 +101,7 @@ def __str__(self) -> str:
 """
 
 _BY_MARKER: dict[str, Capability] = {c.value: c for c in Capability}
+_BY_TAG: dict[str, Capability] = {c.tag: c for c in Capability}
 
 
 def capability_for_marker(name: str) -> Capability | None:
@@ -110,3 +111,14 @@ def capability_for_marker(name: str) -> Capability | None:
     the canonical feature files carry organisational tags freely.
     """
     return _BY_MARKER.get(name)
+
+
+def capability_for_tag(tag: str) -> Capability | None:
+    """Map a Gherkin tag, leading at-sign included, onto the capability it gates.
+
+    The tag form rather than the marker form because that is what the
+    conformance report carries: the report records a scenario's tags as the
+    feature files spell them, and deciding whether a failure counts against a
+    capability means reading them back.
+    """
+    return _BY_TAG.get(tag)

tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py

@@ -73,6 +73,20 @@ def change_flag(self) -> None:
     def description(self) -> str:
         """A short description of what is being controlled, for messages a human reads."""
 
+    # OPTIONAL: ``control_api``
+    #
+    # A control may also offer a ``control_api`` property returning ``"http"``
+    # for the normative HTTP control API, or ``"in-process"`` for the narrow
+    # allowance made for providers with no backend. The conformance report
+    # records it, so that a claim of in-process control by a provider that does
+    # have a backend can be treated with the suspicion it deserves.
+    #
+    # It is deliberately not a member of this protocol. Adding one would make
+    # every existing control incomplete for the sake of one string, and there is
+    # nothing useful the TCK can do with a control that has not said: it cannot
+    # tell from the outside whether a control spoke HTTP or reached into the
+    # process, so the field is simply omitted. See ``report.control_api_of``.
+
 
 @typing.runtime_checkable
 class ConnectionControl(typing.Protocol):