feat(provider-tck): add the Python conformance suite for OpenFeature providers

[aepfli]

Closes #410
Part of open-feature/spec#417 — the
cross-language tracking issue. The language-agnostic artifacts are in
spec#423, their normative description in
spec#439; the report envelope in
spec#425. Java is the reference
(java-sdk-contrib#1830).

Draft — opened for review of the approach. Built and run locally throughout; ruff and
mypy --strict clean. The open questions are where I would most value pushback.

The PR chain

The same four-branch shape in every language, so understanding one is understanding all four:

main
 └── #409  feat/provider-tck            this PR — the suite, both control paths, the compose
      │                                  harness, the extension point, and the entire
      │                                  adopter-facing API
      ├── #413  feat/provider-tck-report    Cucumber Messages and the report envelope
      └── #411  feat/provider-tck-flagd     flagd adoption, both resolvers
           └── #414  feat/provider-tck-ofrep   OFREP adoption

report and flagd are siblings, not a sequence: the follow-ups fill gaps without changing the
adopter-facing API, so neither blocks the other and either can land first.

What this is

A conformance suite any Python provider can adopt to check that it implements the provider contract
in the specification — the Python implementation of Appendix F, running the same
Gherkin scenarios, against the same canonical flag set, driven through the same control API as every
other language's TCK. That shared basis is the point: "conformant" only means something if the
question is identical everywhere.

It uses pytest-bdd, the same runner openfeature-provider-flagd and
openfeature-flagd-api-testkit already depend on, so an adopting package gains no new test
framework.

The adoption surface

Two fixtures and one call:

import pytest
from pytest_bdd import scenarios

from openfeature.contrib.tools.tck import (
    Capability, ComposeBackend, RunningBackend, TckConfig, feature_paths,
)


@pytest.fixture(scope="session")
def compose_backend():
    return ComposeBackend(
        compose_file="tests/tck/docker-compose.yaml",
        backend_ports=[8013],
    )


@pytest.fixture(scope="session")
def tck_config(tck_backend: RunningBackend):
    return TckConfig(
        name="my-provider",
        control=tck_backend.control,
        new_provider=lambda: MyProvider(
            host=tck_backend.endpoint.host,
            port=tck_backend.endpoint.port(8013),
        ),
        capabilities={Capability.EVENTS, Capability.OBJECT},
    )


scenarios(*feature_paths())

No conftest.py, and nothing to import for the steps. The vocabulary ships as a pytest plugin
registered through a pytest11 entry point. That is the nicest adoption of the four, and it is not
an accident — pytest-bdd resolves steps through the fixture system, and fixtures from an installed
plugin are visible to every test.

The suite owns the container stack. You name a Compose file and the ports the provider connects
to; starting Compose, discovering dynamically mapped host ports, building the HTTP control, waiting
until it accepts commands and tearing down are the suite's job. Conventions with working defaults:
backend_service (backend), control_port (8080), additional_ports, backend_configuration
(default), startup_timeout (60s). This is now required by Appendix F, after three of four
languages originally shipped only the control-API client and left orchestration to the adopter — at
which point every adoption hand-rolled the same wrapper.

A provider with no backend supplies its own BackendControl instead and needs no Compose file
and no container tooling: testcontainers is an optional extra, imported lazily. The feature files
and canonical flag set are packaged with the distribution, so adopting needs no git submodule.

Declaring what a provider can and cannot do

An unrecognised canonical tag now fails collection. Appendix F requires it. An unknown tag gates
nothing, so every scenario carrying one stays mandatory for every adopter — a package that has not
learned a new capability does not report a new capability, it silently keeps demanding the old
behaviour. Only canonical tags are read; an adopter's extension tags gate nothing by design, and
reading them as a vocabulary would fail every run that has an extension.

The revision check no longer skips when it cannot read the pin. It used to, and the environment
where the pin was unreadable was exactly the one where a rebase leaves the assets stale — so the check
went quiet where it was load-bearing. The cause turned out to be resolvable rather than fundamental: a
worktree records its git directory as an absolute path, which WSL cannot follow, while a submodule's
is relative, so the pin reads fine from the submodule side. The condition is gone rather than merely
reported. An unpacked distribution is deliberately still exempt: it has no pin and no working tree
that could have drifted, which is a check with nothing to check rather than one that cannot run.

capabilities names the optional parts of the contract this provider supports, from sixteen
members — EVENTS, LIFECYCLE, STALE, CONFIGURATION_CHANGE, OBJECT, VARIANTS,
DISABLED_FLAGS, UNAVAILABLE_INIT, NUMERIC_COERCION, LARGE_INTEGERS, REINITIALIZATION,
TARGETING, STANDARD_REASONS, STRING_TYPING, FULLY_TYPED_VALUES, and CACHING, which is
reserved and cannot be declared. A
scenario gated on an undeclared capability is reported skipped with its reason, never passed:

SKIPPED provider does not declare capability @stale.
        Declared: @events @object @numeric-coercion

known_deviations is the separate statement that the provider fails something it is required to do,
in either a tracked or an untracked form. Appendix F settles which of its two shapes to prefer:
declare the capability, let the scenario fail, and record the deviation beside the failure — rather
than withdrawing the capability so the scenario skips, which hides a defect behind something that
looks deliberate.

reason is asserted in two places only, and they mean different things. Requirement 2.2.5 is a
SHOULD permitting "some other string", so the scenarios that resolve a value no longer assert a
reason at all — an exact-match assertion there fails providers the specification permits. What
replaced it is a capability. STANDARD_REASONS declares that this provider speaks the
specification's own vocabulary, and a feature file then checks the whole of it: STATIC,
TARGETING_MATCH, DEFAULT, DISABLED. Undeclared, those scenarios skip with their reason, and
nothing else in the suite cares what a provider calls a reason.

The error cases are not part of that bargain and are asserted unconditionally. Requirement 1.4.8
makes an error code mandatory in abnormal execution, so ERROR there is not a dialect — a reason
that disagrees with the error code shipped beside it is a contradiction. The assertion is on the two
agreeing, not on which of them is authoritative.

Four things probed rather than assumed

Each verified against pytest-bdd 8.1 before the design depended on it:

Question	Answer
Does pytest-bdd turn Gherkin tags into markers, including dashed ones like @configuration-change?	yes — getattr(pytest.mark, tag) handles them
Does pytest.skip() from an autouse fixture report skipped with the reason?	yes, natively — no reporting machinery needed, unlike Go
Does scenarios() accept an absolute path into an installed package?	yes
Do step definitions work from a plugin rather than the test module?	yes — this is what removes the conftest.py

Two things that only showed up by running it, both fixed and commented:

• pytest-bdd creates markers without registering them, so every tag raised
  PytestUnknownMarkWarning — noise at best, a hard failure under -W error. The plugin registers
  them in pytest_configure.
• The capability gate silently did nothing when guarded on request.fixturenames. pytest-bdd
  resolves a step's fixtures lazily as each step runs, so tck_config is not in fixturenames at
  setup time, and @unavailable scenarios ran against a config that never declared it. The gate now
  keys off the node's markers, which are on the item itself. That is exactly the failure mode the
  suite exists to prevent — a gate that looks right and quietly passes everything — so it is pinned
  by a test.

Self-tests, no Docker

Suite	Subject
test_in_memory_conformance	the SDK's InMemoryProvider — reference adoption for a backend-less provider
test_controllable_conformance	ControllableInMemoryProvider — the configuration-change and lifecycle paths
test_in_process_control	InProcessControl — pins what the Gherkin cannot assert about itself
test_http_control	the HTTP client's request sequence — /reset preferred, /start fallback cached once per suite, /start after a disconnect

There is no multi-provider suite because Python has no multi-provider — worth noting as its own gap.

Findings against the SDK

A boolean satisfies an Integer request, and this is Python-only. boolean-flag through
get_integer_details returns True, reason STATIC, no error code, where the specification
requires the code default and TYPE_MISMATCH. bool subclasses int, so isinstance(True, int)
is True and the type check passes. The identical scenario passes in Java and Go — no suite in
another language could have caught this, which is a fair advertisement for the
multiple-implementations argument.
python-sdk#619, fixed on main and not yet
released, so the row stays xfail(strict=True): it remains visible in the report and fails the
moment it starts passing, which forces the marker's removal when a release carries the fix.

The in-memory provider cannot update its flag set. Appendix A requires it. Same
class of gap as go-sdk#530, found independently
in a second SDK. Only half the machinery is missing — AbstractProvider already supplies
emit_provider_configuration_changed — so ControllableInMemoryProvider is a small subclass
rather than a reimplementation. python-sdk#620,
also fixed on main and unreleased.

A pattern worth naming, because it decides what this suite can currently assert: three of the
findings across two SDKs are fixed on main and carried by no release. #619 and #620 were closed on
2026-08-30; the newest python-sdk release is v0.10.0 from 2026-06-01, so this package resolves a
pre-fix SDK. go-sdk#552 is in the same state.
Each will need a pin bump and a marker removed, not further investigation.

InMemoryFlag.state is never read, so a flag constructed DISABLED resolves as though enabled.
Filed as a question rather than a bug, since carrying the field for configuration compatibility is a
defensible answer: python-sdk#627. The
in-memory suites withhold @disabled-flags on the strength of it.

Two findings in the flagd provider, filed as questions

Both surfaced in this repository's own provider rather than the SDK, and both are filed for triage
rather than asserted as defects:

• shutdown() leaves the gRPC connectivity watcher running, so a clean shutdown emits
  Cannot invoke RPC: Channel closed! after the call has returned
  (#419). Worth noting the
  scenario passes — shutdown completes within its bound — so the results record no failure and the
  report has nowhere to put this. It was found only because the @lifecycle capability had never
  been declared in this adoption, so those six scenarios had never run; declaring it was a one-line
  change.
• The two resolvers disagree on numeric coercion
  (#420): in-process returns
  TYPE_MISMATCH for 0.5 requested as an integer, RPC returns 0 with no error code. Both
  resolvers declare @numeric-coercion; the known deviation is recorded against the RPC suite only,
  because in-process does not have the defect. Calibration worth having: Java's and Go's flagd
  providers narrow in both resolvers, so this provider's in-process path is ahead of the other
  implementations rather than behind them.

Running it

The gating half of this is provisional and under discussion. The suite is excluded because its
honest output is red — but that is only true while a run is judged by "zero failures". Judged
instead by whether its results match its declaration — every failure covered by a declared
deviation, every skip gated by an undeclared capability, every deviation still failing something —
a healthy adoption is green in its steady state, deviations included, and this could be a required
gate. Proposed in
open-feature/spec#417;
nothing here depends on the outcome. What is settled either way is that the conformance suite gets
a step of its own rather than being folded into the provider's e2e suite, so a red build says
conformance failed rather than a test failed.

The self-tests run in the default build with no Docker. The adoption suites live in tests/tck/,
a sibling of tests/e2e/ — Python was already the shape the other three moved to — and the directory
is the whole of the selection: --ignore=tests/tck excludes them from poe test/poe cov,
pytest tests/tck runs them. No -k, no python_files override, no name-shaped glob anywhere in
the chain, which is why the three adoption modules could drop the word conformance from their names
without a single count moving.

poe test-tck

The default build still typechecks the adoption, which Appendix F requires: --ignore leaves the
files in the tree and in the package, and poe test ends with a second pytest invocation that
collects tests/tck without running it. That step cannot pass having checked nothing — pytest exits
5 on an empty collection and 4 on a path that does not exist, measured four ways — so no count
assertion is needed on top of it. It was, however, not being reached on flagd: poe aborts a
sequence at its first failing subtask, and flagd's default suite is red on this stack for the SDK
reason below, so the check the README described had never run in the package where it matters most.
Both adoptions now set ignore_fail = "return_non_zero", which runs every subtask and still
propagates a non-zero exit.

Appendix F carries the reasoning: an adoption suite's honest output is red while real gaps remain,
so making it a required gate forces someone to silence it — and the cheapest way to silence a
conformance suite is to stop asking the question. Note Docker is not the reason: tests/e2e
needs Docker too and still runs in the default build.

So read the tally below rather than the green check, on the adoption PRs. Worth stating plainly:
this suite previously ran in the default build with nothing excluding it, red, unnoticed. All four
languages had a version of that, each defeated by a different mechanism.

Verification

At 01bb657f:

Check	Result
poe test (this package)	221 passed, 42 skipped, no Docker
ruff check, mypy via poe	clean
Assets copied from the spec submodule at build time	yes — no committed copy to drift
flagd adoption, run deliberately	119 passed, 3 skipped, 8 failed
OFREP adoption, run deliberately	45 passed, 17 skipped, 2 failed, 1 xfailed

The flagd failures are documented backend fixture gaps plus the isinstance(True, int) consequence
in the flagd provider's own type check; the OFREP ones are
python-sdk-contrib#418 — variant
and reason indexed unconditionally on a response where the OFREP schema makes both optional.

A red flagd job on this PR, which is not this PR's suite

Worth reading before the CI result: the flagd job is red on this branch and tests/tck is not
why.

openfeature-tck requires openfeature-sdk>=0.10.0. A uv workspace resolves one version per
member, so uv.lock moves from 0.8.4 to 0.10.0 across the workspace — and 20 of the flagd
provider's existing tests fail on 0.10.0. Measured both ways in the same venv: 20 failed, 9 passed at 0.10.0, 29 passed after installing 0.8.4 back. build.yml's path filter lists
uv.lock for every package, so this PR runs the flagd job and that job goes red on a branch that
touches no flagd source.

There is a second casualty, and an earlier revision of this description wrongly said there was
not. unleash fails poe mypy on 0.10.0: its track() is
track(self, event_name, event_details) where the SDK's AbstractProvider.track is now
track(self, tracking_event_name, evaluation_context, tracking_event_details), so the override is
incompatible. It surfaces only on the 3.14 job because that is TARGET_PYTHON_VERSION, the only one
that runs poe mypy — which is why a test-only sweep missed it. The remaining packages — aws-ssm,
env-var, flipt, flagd-api, flagd-api-testkit and flagd-core — do pass.

Both casualties are the same kind of change and neither is fixed here: they are provider migrations
to a newer SDK, and they want their own PRs and their own reviewers.

Deliberately not fixed here: it is a provider migration to a newer SDK, and it wants its own PR
and its own reviewer. Flagging it because a reviewer seeing red on this branch would reasonably
assume the conformance suite caused it.

Known gaps

• Evaluation context passthrough beyond the targeting key. targeting-key-flag catches a
  provider that drops the context entirely; a provider that forwards the targeting key and silently
  discards every other attribute still passes. Closing it needs an echo operation on the control
  API, or a canonical flag whose rule keys on a custom attribute.
• Targeting and bucketing correctness — out of scope by design; that is backend evaluation logic.
• Caching — @caching is reserved, no scenarios yet.
• Per-flag set/remove control operations — would need control API endpoints that do not exist.
• Hooks — not covered.
• No multi-provider suite, because the Python SDK has no multi-provider.

Open questions

1. Nothing validates an emitted report against the schema in CI, in any language. The report
   branch checks fields by hand and validates out of band. The schema is what makes reports
   comparable, so this is the weakest link in the design and it wants one answer rather than four.
2. Should ControllableInMemoryProvider live here at all, or should the SDK fix land first and
   this package depend on it?

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 86.05919% with 179 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.98%. Comparing base (574abf2) to head (1d03e75).

Files with missing lines	Patch %	Lines
.../openfeature/contrib/tools/tck/steps/flag_steps.py	75.96%	31 Missing ⚠️
...e-tck/src/openfeature/contrib/tools/tck/compose.py	77.77%	30 Missing ⚠️
...openfeature/contrib/tools/tck/steps/event_steps.py	61.19%	26 Missing ⚠️
...nfeature/contrib/tools/tck/steps/provider_steps.py	70.93%	25 Missing ⚠️
tools/openfeature-tck/hatch_build_sync.py	78.49%	20 Missing ⚠️
...ure-tck/src/openfeature/contrib/tools/tck/state.py	86.91%	14 Missing ⚠️
...re-tck/src/openfeature/contrib/tools/tck/values.py	80.35%	11 Missing ⚠️
...re-tck/src/openfeature/contrib/tools/tck/config.py	93.80%	7 Missing ⚠️
...ck/src/openfeature/contrib/tools/tck/extensions.py	96.34%	3 Missing ⚠️
...k/src/openfeature/contrib/tools/tck/httpcontrol.py	97.58%	3 Missing ⚠️
... and 6 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #409      +/-   ##
==========================================
- Coverage   95.34%   88.98%   -6.37%     
==========================================
  Files          24       44      +20     
  Lines        1075     2215    +1140     
==========================================
+ Hits         1025     1971     +946     
- Misses         50      244     +194     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.