feat(provider-tck): emit a conformance report whose results are Cucumber Messages

tools/openfeature-tck/.gitignore

@@ -5,3 +5,6 @@
 src/openfeature/contrib/tools/tck/gherkin/
 src/openfeature/contrib/tools/tck/flag_data/
 src/openfeature/contrib/tools/tck/control-api.yaml
+# Generated alongside them, from the submodule pin, so a conformance report can
+# name the spec revision it ran against.
+src/openfeature/contrib/tools/tck/spec_revision.json

tools/openfeature-tck/README.md

@@ -312,6 +312,83 @@ self-tests declare less than they otherwise would: it cannot update its flag set
 hands each variant back untouched, so it does not attempt numeric coercion at all — a permitted
 choice rather than a defect, and the reason `@numeric-coercion` is simply not declared there.
 
+## Conformance reports
+
+Set `TCK_REPORT_DIR` and each suite writes **two** files: an envelope at `<dir>/<name>.json`,
+conforming to the [report schema][report-schema], and the results it points at, at
+`<dir>/<name>.ndjson`, which is a [Cucumber Messages][messages] stream.
+
+```console
+$ TCK_REPORT_DIR=./reports pytest
+tck [in-memory]: report written to reports/in-memory.json with results in in-memory.ndjson (1 failed, 43 passed, 21 skipped)
+
+$ jq -c .results reports/in-memory.json
+{"format":"cucumber-messages","location":"in-memory.ndjson","digest":"sha256:c7e12a…"}
+```
+
+It is an environment variable rather than a `TckConfig` field so that emitting a report is a property
+of the *run* and not of the code: CI sets it, a developer running the suite locally does not, and no
+adopter changes a line to publish one. Unset means no report, which is not an error, and several
+suites in one session each write their own pair, so flagd's two resolvers do not collide.
+
+**A partial run is not a conformance run.** `-k`, `-m`, `--deselect`, or a test module that stopped
+calling `scenarios()` on the canonical path each run fewer scenarios, and none of them is an error to
+pytest — Go measured the consequence: a green run and a well-formed report covering one scenario out
+of the whole canonical set, with nothing in the document saying so. So every run is checked against
+the scenarios this distribution ships, and one that did not execute all of them writes no report and
+names what it missed. A capability-gated skip **has** run — the question was put and declined — so
+declining never trips this, and your own extension scenarios are not counted towards the canonical
+set and cannot close a gap in it. `TCK_PARTIAL=1` lets you work on a single scenario without the
+guard failing the run; it still writes no report, and Java spells the same escape hatch the same way.
+
+**Why the results are not our format.** Per-scenario outcomes, tags, Scenario Outline row identity
+and the executed feature source are already specified by Cucumber Messages, which is maintained,
+cross-language, schema'd and emitted natively by cucumber-jvm; defining them again would create a
+second format to version and two places for the same fact to disagree. So the envelope says what was
+tested and what the provider claims, and the payload says what happened. The payload is referenced
+rather than inlined because it carries the feature sources and is far larger than the envelope, and
+`results.digest` is a SHA-256 over the exact bytes written, so a consumer can tell that what it
+fetched is what the envelope described. Two things Messages cannot carry stay in the envelope:
+`declaration`, which is an *input* to reading the results rather than a summary of them — only it
+says whether a skipped scenario was declined — and the tested subject, for which no standard results
+format has a slot.
+
+**Reading the payload.** Appendix F requires a scenario skipped for an undeclared capability to be
+reported as skipped with the reason and never as passed, and a consumer cannot check that against a
+summary line, so the stream carries every scenario the run collected, gate-skipped ones included, as
+Cucumber's own `SKIPPED`. Each is a `TestCase` referring to a `Pickle`, and a test case is as bad as
+its worst step. Every test case carries two hook steps as well as its Gherkin steps, because pytest
+runs a scenario in three phases and only the middle one executes steps: the before-hook is where a
+capability skip's reason lands and the after-hook where a teardown failure does. The capability
+responsible for a skip follows from the pickle's tags and the envelope's `declaration`, which is why
+it is not transported once per scenario. And a pickle's `astNodeIds` are `[scenario id, table row
+id]`, resolving in the `GherkinDocument` to exactly the cells the feature file wrote — which is what
+tells the eight rows of the type-mismatch matrix apart, one of which differs in outcome from its seven
+siblings, exactly rather than by a naming convention every implementation would have to reproduce.
+
+So the payload is not a transcription of pytest's summary. The one scenario the Python SDK cannot
+satisfy is marked `xfail`, so pytest counts it as expected and exits zero; the provider still did not
+satisfy it, and the stream says `FAILED`. The acknowledgement goes in the envelope's
+`knownDeviations` instead — an expected failure is a recorded deviation, not an excused one.
+
+**What identifies a report.** `tck.specRevision` comes from `spec_revision.json`, which the asset
+sync generates from the submodule at build time, because the submodule is not in the wheel and an
+installed copy has nothing left to ask; a build that cannot reach git records `unknown` rather than
+inventing a commit. There is no asset tree hash: the payload's `Source` messages carry the executed
+feature files verbatim, which answers "did these two runs ask the same questions" directly rather
+than by proxy. `provider.name` is what the provider reports through its own metadata, not
+`TckConfig.name`, which is chosen to read well in a failure message — `flagd-rpc` — and is therefore
+reported as the *configuration*. `backend.controlApi` is read straight off the required `control_api`
+member and the whole `backend` block is always written, both being in the schema's `required` arrays:
+there is nothing to fall back to and nothing inferred, which is the point.
+
+One gap is this implementation's rather than the suite's: **pytest-bdd emits no Cucumber Messages.**
+It ships the legacy Cucumber JSON format and nothing for the ndjson protocol, so `messages.py` builds
+the stream from the official types and re-parses the feature files for the AST node ids a pickle
+refers to. If pytest-bdd ever emits Messages itself, that module should shrink to a shim. Whether a
+report belongs inside a provider's released artifact is open on
+[open-feature/spec#424](https://github.com/open-feature/spec/issues/424).
+
 ## Contributing
 
 The Gherkin, the canonical flag set and the control-API document are **not owned by this
@@ -320,7 +397,7 @@ sdist at build time. So adopting needs no submodule and contributing does:
 
 ```bash
 git submodule update --init tools/openfeature-tck/spec
-poe test   # syncs the assets first; 227 passed, 41 skipped, 2 xfailed, no Docker
+poe test   # syncs the assets first; 295 passed, 41 skipped, 2 xfailed, no Docker
 ```
 
 The copies under `src/` are gitignored, generated and carry a `DO-NOT-EDIT.txt`: a change goes to
@@ -342,6 +419,8 @@ longer defeats it, because the pin is read by naming the superproject that git c
 remains exempt is an unpacked sdist, which has no submodule, no pin and nothing that could have
 drifted from one.
 
+[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json
+[messages]: https://github.com/cucumber/messages
 [appendix-a]: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md
 [appendix-f]: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md
 [spec]: https://github.com/open-feature/spec

tools/openfeature-tck/hatch_build.py

@@ -18,15 +18,28 @@
 # the single definition of what gets copied where -- would not be importable.
 sys.path.insert(0, str(Path(__file__).parent))
 
-from hatch_build_sync import FILES, PACKAGE_REL, SPEC_ASSETS, TREES, sync
+from hatch_build_sync import (
+    FILES,
+    PACKAGE_REL,
+    REVISION_FILE,
+    SPEC_ASSETS,
+    TREES,
+    sync,
+)
 
 
 class SpecAssetsCopyHook(BuildHookInterface):
     PLUGIN_NAME = "spec-assets-copy"
 
     def initialize(self, version: str, build_data: dict) -> None:
         root = Path(self.root)
-        copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES]
+        # The generated revision file travels with the assets it describes. It
+        # has to be built here rather than read at run time, because the
+        # submodule that knows the answer is not in the wheel and a conformance
+        # report has to name the revision it ran against.
+        copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES] + [
+            root / PACKAGE_REL / REVISION_FILE
+        ]
 
         # Building from a checkout: refresh from the submodule, so what ships is
         # always the revision the pin names. Building from an sdist: there is no

tools/openfeature-tck/hatch_build_sync.py

@@ -15,6 +15,7 @@
 
 from __future__ import annotations
 
+import json
 import os
 import shutil
 import subprocess
@@ -25,7 +26,8 @@
 ROOT = Path(__file__).parent
 SPEC_DIRNAME = "spec"
 SPEC_ROOT = (ROOT / SPEC_DIRNAME).resolve()
-SPEC_ASSETS = (SPEC_ROOT / "specification/assets/provider-tck").resolve()
+ASSETS_PATH_IN_SPEC = "specification/assets/provider-tck"
+SPEC_ASSETS = (SPEC_ROOT / ASSETS_PATH_IN_SPEC).resolve()
 PACKAGE_REL = Path("src/openfeature/contrib/tools/tck")
 DEST_BASE = ROOT / PACKAGE_REL
 
@@ -243,6 +245,28 @@ def checkout_pinned_spec(git: GitRunner = _run_git) -> str | None:
     return pinned
 
 
+REVISION_FILE = "spec_revision.json"
+"""Which revision of the specification the copied assets came from.
+
+Recorded at build time because the answer is only available at build time: the
+submodule that holds it is not in the wheel, and a conformance report that cannot
+name the revision it ran against cannot be compared with another. It is generated
+by the same command that copies the assets, which is what keeps the two from
+disagreeing.
+
+Not committed, for the same reason the assets are not: the submodule pin is the
+single record of which revision this package targets.
+"""
+
+UNKNOWN_REVISION = "unknown"
+"""Seven characters, the minimum the report schema accepts.
+
+A build that cannot reach git says it does not know rather than inventing a
+commit, and still produces a document that validates. Which happens for real:
+building from a source tarball has no ``.git`` to ask.
+"""
+
+
 def sync() -> None:
     checkout_pinned_spec()
 
@@ -267,6 +291,48 @@ def sync() -> None:
             dest.unlink()
         shutil.copy2(SPEC_ASSETS / src_name, dest)
 
+    write_revision()
+
+
+def write_revision() -> None:
+    """Record the spec commit these copies came from.
+
+    The asset tree hash that used to accompany it is gone. It was carried so a
+    consumer could tell whether two runs executed the same questions; the
+    conformance report's results are now a Cucumber Messages stream, which
+    carries the executed feature source itself and answers that directly rather
+    than by proxy.
+    """
+    commit = _git("rev-parse", "HEAD") or UNKNOWN_REVISION
+    (DEST_BASE / REVISION_FILE).write_text(
+        json.dumps({"specRevision": commit}, indent=2) + "\n",
+        encoding="utf-8",
+    )
+
+
+def _git(*args: str) -> str:
+    """Run git inside the submodule, returning its output or an empty string.
+
+    A build must not hard-fail because git is absent or the checkout is not a
+    repository -- both are ordinary when building from an unpacked sdist. The
+    failure is reported as a warning and the identity degrades to ``unknown``,
+    which is legible in the resulting report rather than silently wrong.
+    """
+    command = ["git", "-C", str(SPEC_ROOT), *args]
+    try:
+        completed = subprocess.run(  # noqa: S603
+            command, capture_output=True, check=True, text=True
+        )
+    except (OSError, subprocess.CalledProcessError) as error:
+        warnings.warn(
+            f"could not determine the spec revision ({' '.join(command)}: {error}); "
+            f"conformance reports from this build will not name the revision they "
+            f"ran against",
+            stacklevel=2,
+        )
+        return ""
+    return completed.stdout.strip()
+
 
 if __name__ == "__main__":
     sync()

tools/openfeature-tck/pyproject.toml

@@ -22,6 +22,21 @@ dependencies = [
     # Same runner the flagd provider and the flagd testkit already use, so an
     # adopting module gains no new test framework.
     "pytest-bdd>=8.1.0,<9.0.0",
+    # The conformance report's results are a Cucumber Messages stream rather
+    # than a format this package defines. These two are the reference
+    # implementations of the halves of that protocol: cucumber-messages is the
+    # official Python types, published from the same repository as the protocol
+    # itself, and gherkin-official is the parser that produces the
+    # gherkinDocument and pickle messages. pytest-bdd already depends on
+    # gherkin-official, so only the first is genuinely new -- and it has no
+    # dependencies of its own.
+    #
+    # pytest-bdd ships no Messages emitter (its cucumber_json.py is the legacy
+    # JSON format), so the stream is assembled here; assembling it from typed
+    # messages rather than hand-written dicts is what keeps it from drifting
+    # away from the protocol.
+    "cucumber-messages>=34.0.0,<35.0.0",
+    "gherkin-official>=29.0.0",
 ]
 requires-python = ">=3.10"
 
@@ -69,6 +84,10 @@ artifacts = [
     "src/openfeature/contrib/tools/tck/gherkin/",
     "src/openfeature/contrib/tools/tck/flag_data/",
     "src/openfeature/contrib/tools/tck/control-api.yaml",
+    # Which spec revision those assets came from, generated beside them. The
+    # submodule is not in the wheel, so a conformance report emitted by an
+    # installed copy has no other way to name the revision it ran against.
+    "src/openfeature/contrib/tools/tck/spec_revision.json",
 ]
 
 [tool.hatch.build.hooks.custom]
@@ -92,6 +111,21 @@ fixed_format_cache = true
 pretty = true
 strict = true
 disallow_any_generics = false
+# cucumber-messages and gherkin-official ship no py.typed. Both are annotated
+# internally, so following them gives real types for the messages this package
+# builds rather than the Any a plain `ignore_missing_imports` would hand back --
+# which is the point of using the typed library at all.
+follow_untyped_imports = true
+
+[[tool.mypy.overrides]]
+# gherkin-official has no annotations at all, so following it turns every call
+# into a `no-untyped-call` error rather than into a type. pytest-bdd silences
+# the same import the same way. cucumber-messages is the opposite case -- fully
+# annotated, only missing py.typed -- and is followed, which is where the value
+# of using it rather than hand-written dicts actually lands.
+module = ["gherkin.*"]
+follow_untyped_imports = false
+ignore_missing_imports = true
 
 # testcontainers 4.14 ships no py.typed for `testcontainers.compose`, so the one
 # lazy import in compose.py cannot be checked against it. Scoped to that module

tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py

@@ -76,6 +76,7 @@ def tck_config():
 
 import importlib.resources
 
+from .canonical import PARTIAL_ENV
 from .capability import (
     DECLARABLE_CAPABILITIES,
     INEXPRESSIBLE_CAPABILITIES,
@@ -109,12 +110,14 @@ def tck_config():
     HttpControl,
 )
 from .inprocess import InProcessControl
+from .messages import MESSAGES_FORMAT
 from .provider import (
     CHANGING_FLAG_KEY,
     ControllableInMemoryProvider,
     canonical_flag_set,
     canonical_flags_json,
 )
+from .report import REPORT_DIR_ENV, SCHEMA_VERSION
 from .state import TckState
 
 __all__ = [
@@ -126,7 +129,11 @@ def tck_config():
     "DEFAULT_STARTUP_TIMEOUT",
     "EXTENSIONS_DIRECTORY",
     "INEXPRESSIBLE_CAPABILITIES",
+    "MESSAGES_FORMAT",
+    "PARTIAL_ENV",
+    "REPORT_DIR_ENV",
     "RESERVED_CAPABILITIES",
+    "SCHEMA_VERSION",
     "BackendControl",
     "BackendEndpoint",
     "Capability",