Publishing provider conformance: a machine-readable report, and how to collect one from providers we do not host

[aepfli]

Follow-up to #417, which asks in its open questions: "Who owns conformance claims? Is there a badge, a report format, a listing on openfeature.dev? Or is passing simply its own reward for now?" This issue is that question, split out so it can progress on its own timeline — #417 is already carrying Appendix F, four language implementations and a findings table.

This is a design discussion, not a proposal to implement. The positions below are opening arguments, and the two marked as traps are the ones I would most like challenged.

The opportunity

The provider conformance suite (Appendix F, #423) now has implementations in four languages, each of which already computes exactly the data a public conformance report would need:

• the capability set a provider declares
• a per-scenario outcome, with a reason string attached to every skip

What is missing is an agreed report format and a way to publish and collect reports. That would let openfeature.dev show, for any provider, what it actually supports — replacing "implements the provider contract" as an unverified claim with something a user can look up before choosing a provider.

The awkward part

Not every provider lives in an SDK-contrib repo. Many are in vendor organisations, on their own release cadence, and OpenFeature cannot run their suites — the backends and credentials are not ours. So any design has to work for providers we do not host and cannot execute.

Three problems, hardest last

1. Comparability

A report means nothing unless you know it answers the same question as the report beside it. Each report must carry the TCK implementation, its version, the spec revision its assets came from, and a digest of the Gherkin and canonical flag set actually executed. Without that last one, two reports can differ for unknowable reasons — the exact failure this whole effort exists to prevent.

These are not independent facts. The spec revision is a property of the TCK artifact, not something the report asserts separately: a given TCK build vendors exactly one revision. So they collapse into a single object rather than sitting in a separate assets block.

The revision and the digest are still worth carrying together, because they do different jobs. The revision is an assertion; the digest is a check. A recorded revision can be wrong — someone edits a vendored feature file and the label still points at the original commit. The digest catches precisely that, which is the drift this whole effort exists to prevent. They are a self-checking pair, not duplicates.

This has been settled at the source rather than in the schema. The decision is that no TCK vendors the conformance assets: every implementation consumes them from this repository as a git submodule, so the spec revision is always recorded by the submodule pin. Java already did this; Go, Python and JavaScript are being converted.

Two consequences for this issue:

• specRevision is never unknown and never has to be reconstructed — it is whatever the submodule points at, available to the emitter for free.
• An adopter of a TCK still needs no submodule of their own. Each implementation packages the assets into its published artifact (embedded in the Go package, in the Python wheel, in the JAR, in the npm package), so the submodule is a concern for TCK contributors only. That property is worth stating in the schema documentation, because it is what stops "conformance report" from implying "clone two repositories".

2. Trust

A JSON file in a vendor repository is a claim, not a fact. Three options:

Approach	Verdict
OpenFeature re-runs every suite centrally	does not scale — needs each vendor's backend and credentials
Naked self-report	worthless the moment anything is contested
Self-report with provenance	the only one that scales

Provenance meaning: the report is emitted only by CI, and carries the workflow run URL, commit SHA, and optionally a sigstore / GitHub OIDC attestation. This is what the rest of the supply-chain ecosystem already does for build artifacts.

Whatever we choose, the page should say plainly what it is showing: attested self-reports, not audits.

3. Discovery

Do not crawl. Suggested shape:

• a registry of pointers in an OpenFeature-owned repo — provider → report URL, added once by a vendor PR
• vendors self-publish the report to that URL (release asset, GitHub Pages, wherever)
• a scheduled job fetches every registered URL, validates against the schema, and rebuilds the page

The human-reviewed list stays small and stable; report content refreshes without anyone raising a PR. A /.well-known/openfeature-conformance.json convention could layer on later for vendors who prefer it.

Trap 1: two outcomes is not enough

A report needs four outcomes:

Outcome	Meaning
passed	ran, conformed
failed	ran, did not conform
not-declared	the provider says it does not do this
not-applicable	the language cannot express the question

That fourth category is not hypothetical — it appeared while building the JavaScript implementation. @strict-numeric-typing asserts that a provider reports TYPE_MISMATCH rather than narrowing 0.5 to 0 when a float flag is requested as an integer. JavaScript has no integer type: typeof 10 and typeof 0.5 are both 'number', and the Evaluation API exposes only getNumberDetails. Requesting float-flag as an Integer is indistinguishable from requesting it as a Float, so no JavaScript provider can ever satisfy that scenario — not through a defect, but because the distinction does not exist in the language.

Collapse not-applicable into not-declared and the overview page will permanently show every JS provider as missing a capability no JS provider can have. That is a defect in the report rendered as a defect in the provider.

(This also feeds back into #417's open question about whether @strict-numeric-typing should be language-scoped at the specification level.)

Trap 2: never compute a conformance percentage

This is the one I feel strongest about.

A single score rewards declaring fewer capabilities. Undeclare @stale and your percentage goes up. That is conformance theatre, and it exactly inverts the incentive the suite exists to create.

Instead: a matrix of provider × capability, four distinct states, no aggregate number, and the skip reason shown. "This provider does not support configuration-change events" is genuinely useful to someone choosing a provider — hiding it behind a green 100% is actively harmful. not-declared should be as visible as failed.

Sketch of a report

Illustrative only — the field names are the least interesting part of this issue:

{
  "schemaVersion": "1",
  "provider": { "name": "flagd", "version": "0.14.1", "language": "java", "configuration": "rpc" },
  "sdk": { "name": "dev.openfeature:sdk", "version": "1.17.0" },
  "tck": {
    "implementation": "java-sdk-contrib/tools/provider-tck",
    "version": "0.1.0",
    "specRevision": "42b047b9",
    "gherkinDigest": "sha256:..."
  },
  "backend": { "description": "flagd-testbed v3.10.1", "controlApi": "launchpad" },
  "provenance": { "runUrl": "https://github.com/.../actions/runs/123", "commit": "abc123" },
  "capabilities": {
    "events": "passed",
    "stale": "not-declared",
    "strict-numeric-typing": { "state": "not-applicable", "reason": "no integer type in this language" }
  },
  "scenarios": [
    { "feature": "errors", "name": "Requesting the wrong type returns the code default", "outcome": "passed" }
  ]
}

Note provider.configuration: one provider can produce several reports. flagd has two resolvers with genuinely different conformance — its Go RPC resolver does not emit PROVIDER_STALE while its in-process one does (go-sdk-contrib#939). A report keyed only on provider name would have to pick one and lie about the other.

Suggested phasing

1. Schema in this repo, next to the Gherkin — same kind of artifact, must version alongside it.
2. Emitters in each language TCK. The data already exists; this is a JSON writer.
3. Registry + aggregator, designed once two or three real reports exist.
4. The page, last.

Steps 1–2 are worth doing soon because they are cheap and they make step 3 designable against real data. Steps 3–4 should not be designed in the abstract.

Questions

1. Should every TCK be required to record the spec revision it vendored? Settled: assets are consumed from this repository via git submodule in every implementation, so the revision is always recorded by the pin. The open part is narrower — should the report carry the submodule commit, a spec release tag, or both, given that a commit is precise but a tag is what a human can reason about?
2. Is not-applicable the right fourth state, or should the specification instead scope capabilities per language so the situation cannot arise?
3. Registry-of-pointers, or a well-known URL convention, or both?
4. How much provenance is proportionate — a CI run URL, or a full signed attestation?
5. Does a report expire? A report generated against a two-year-old TCK revision is not obviously worth displaying, but silently dropping it is its own kind of dishonesty.
6. Who owns the registry and the aggregator — this repo, openfeature.dev, or something new?

[aepfli]

Phase 1 is now a PR: #425

The report schema is up as #425, stacked on #423. It was previously sitting inside #423 itself, which was wrong — the report format is a separate decision from the appendix and deserves its own review.

It covers only the part of this issue that seemed settled: what a report must contain for two reports to be comparable. Collection and trust stay open here.

The decisions most worth arguing with are called out in the PR description: four outcomes rather than two, per-scenario results required and required to be complete, and deliberately no aggregate score.

One update from running the suites, relevant to knownDeviation

The case for a distinct known-deviation concept has got stronger since this issue was written. It is now three implementations that each invented their own local shape for it:

• flagd narrows a float to an integer, in both Java and Go, in both resolvers of each — @strict-numeric-typing withheld in both adoptions.
• Python uses xfail(strict=True) for A boolean flag satisfies an Integer request: bool is a subclass of int python-sdk#619, so the deviation stays visible and un-hides itself when the SDK is fixed.
• JavaScript's multi-provider (the one in @openfeature/server-sdk, not the deprecated contrib package) replaces every child error code with GENERAL — 15 × TYPE_MISMATCH and 1 × FLAG_NOT_FOUND out of 29 scenarios. Everything else survives delegation intact.

That last one is worth noting for a reason beyond the count: it is not capability-gated and cannot be. Reporting the right error code is not an optional capability, so there is no tag to withhold, and the only options available today are "exclude the suite" or "ship a red build". Both lose the information. A report format that can say failed, known deviation, tracked here is the thing that would let it be published honestly instead of hidden.

Phase 2, emitters

Not started, deliberately. Writing four emitters against a schema still under discussion means rewriting four emitters. They follow agreement on the shape rather than preceding it.

[aepfli]

Phase 2 has started: the first emitter is up

go-sdk-contrib#944 emits this format from the Go TCK. Its reports validate against the schema in #425 with a Draft 2020-12 validator, which is the first evidence the schema is implementable rather than merely plausible.

Go was taken first deliberately, because it is the implementation the format most has to justify itself against. Concretely:

29 scenarios (29 passed)      <- godog's summary

Five of those twenty-nine did not run — they were skipped for undeclared capabilities. The emitted report for the same run says {"passed": 24, "not-declared": 5}, with a reason on each skip.

That is the argument for requiring per-scenario results, made concrete: the report does not fix the runner's summary, it makes the summary stop mattering. A consumer can check "a skipped scenario is never reported as passed" for themselves rather than trusting each runner to have applied it. pytest and jest-cucumber happen to report skips correctly, so without this the rule would hold in three languages and quietly fail in the fourth.

Two things the implementation settled

assetsTree is free in practice. The Go TCK already had a generator that copies the artifacts out of the submodule under a CI check; recording the revision and tree there was a few lines, and the existing check keeps them honest — a revision that disagrees with the artifacts beside it cannot be committed. This is the concrete reason the content digest was dropped in favour of the tree hash.

provider.name should be the provider's own metadata name, not the suite's. The suite name is chosen to read well in failure messages (flagd-rpc), which makes it the configuration. Worth stating in the schema documentation, because getting it backwards would silently produce reports that look provider-scoped and are not.

Still open here

Collection and trust — how OpenFeature receives reports from providers it does not host, and how far a self-published report should be believed — are untouched by either PR and remain the substance of this issue.

[aepfli]

Three emitters exist now, and they disagree about one thing

go-sdk-contrib#944, python-sdk-contrib#413 and js-sdk-contrib#1610. All three emit reports that validate against the schema in #425 under a Draft 2020-12 validator. Java is in progress.

Implementing the format turned out to be the real review of it. Three problems were clear enough to fix directly in #425 — a reason is now enforced for any outcome other than passed rather than merely expected; knownDeviation is available on a scenario as well as a capability; and capabilities now states that it is not a verdict. The fourth needs a decision here.

Scenario Outline rows are not portably identifiable

scenarios[] is keyed on (feature, name), and the schema assumes that pair is unique. It is not. Every row of a Scenario Outline shares one scenario name, and there is no field for the row.

This is not hypothetical. In the Python run, one row of "Requesting the wrong type returns the code default" has a different outcome from its ten siblings. Reported under a shared name, ten rows say passed and one says failed under the same key — a consumer cannot tell which case failed, and a naive one that builds a map keyed on (feature, name) silently keeps whichever row it saw last.

The three implementations have each solved it differently, because nothing told them not to:

	how an outline row is identified
Go	godog's own expansion of the scenario name
Python	the scenario name qualified with pytest's example id, e.g. … [boolean-flag-Integer-1]
JavaScript	jest-cucumber's expanded example title

All three are locally reasonable and none of them agree, so the same row of the same feature file appears under three different keys in three reports. Comparability across languages is the entire justification for having a shared format, and this is the one place it currently fails.

Worth noting that JavaScript hit a second, related problem: jest-cucumber never applies a scenario-name template to outline examples, so on the base branch four skipped outline rows were reported with no reason at all. Different bug, same root — outline rows are the case every runner treats as an afterthought, which is a good argument for the format being explicit about them rather than inheriting whatever each runner does.

Options

1. Add an optional example field to scenarioResult carrying the row's parameters as an object ({"flag": "boolean-flag", "type": "Integer"}), with the key becoming (feature, name, example). Precise and language-neutral, since the parameters come from the feature file rather than from the runner. Costs each implementation a little plumbing to get the example row out of its runner.
2. Add a required rowIndex — cheaper, but positional, so inserting a row into an Examples table silently re-keys every row after it.
3. Mandate a name-mangling convention, e.g. name [param1=value1, param2=value2]. No schema change, but it puts a string format in the normative text and every implementation has to reproduce it exactly, which is the kind of thing that drifts quietly.

I lean towards option 1: the parameters are the identity, and putting them in a field rather than smuggling them into a string is what lets a consumer group, diff and display them. It also makes the "one row of eleven failed" case legible, which is the case that exposed the problem.

Happy to be argued out of it — option 3 is meaningfully less work for the implementations, and if the view is that reports are read by humans more than by tooling, that changes the calculus.

[aepfli]

Decided: a Scenario Outline row is identified by its parameters

scenarios[] entries were keyed on feature + name, and every row of an outline shares one name. The type-mismatch matrix is eleven rows, so a report carried eleven entries separated by nothing but a duration measurement — and the Python implementation produced exactly the case that makes this matter, with one of the eleven failing and ten passing under one key.

An entry now carries the Examples row it came from, keyed by column header:

{
  "feature": "errors",
  "name": "Requesting the wrong type returns the code default",
  "example": { "key": "boolean-flag", "requested": "Integer", "default": "1" },
  "outcome": "failed",
  "reason": "error-code was 'GENERAL', expected 'TYPE_MISMATCH'"
}

Identity is (feature, name, example). Pushed to #425.

Why a field rather than a naming convention

The alternative was mandating a mangled name — name [key=boolean-flag, requested=Integer, default=1] — which needs no schema change and less work per implementation. It was rejected because it puts a separator, an ordering, and an escaping rule for values containing the separator into normative prose that every implementation has to reproduce byte-for-byte, and drift there stays invisible until two reports quietly fail to line up.

That is not a hypothetical. The three implementations had already diverged on this point before the field existed, each locally reasonable and none compatible with the others:

	what landed in name for the same row
Go	the bare scenario name, for all eleven rows
Python	the name plus pytest's example id
JavaScript	jest-cucumber's expanded example title

Values are the cell contents verbatim as strings, because Gherkin has no types — coercing "1" to a number would make the report disagree with the table it came from.

Related, and worth noting for anyone writing an emitter

JavaScript hit a second outline problem: jest-cucumber never applies a scenario-name template to outline rows, so four skipped rows were reported with no reason at all — a direct violation of Appendix F's "skipped, with the reason". Different bug, same root cause: outline rows are the case every runner treats as an afterthought. That is a good argument for the format being explicit about them rather than inheriting whatever each runner happens to do.

All four implementations are being updated to populate the field.