Skip to content

feat!: replace root module with the black-box SDK test harness - #13

Merged
TomChv merged 7 commits into
mainfrom
sdk-test
Aug 5, 2026
Merged

feat!: replace root module with the black-box SDK test harness#13
TomChv merged 7 commits into
mainfrom
sdk-test

Conversation

@kpenfound

@kpenfound kpenfound commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What

Replaces the root sdk-sdk module with the black-box SDK test harness, carrying forward the still-useful tooling from the previous root module. The root module does it all:

dagger -m github.com/dagger/sdk-sdk -W <sdk-repo> check

Lifecycle checks

The checks vendor the SDK module into a scratch git workspace inside a runner container (pinned, checksum-verified release CLI, default 1.0.0-beta.9) and drive it the way a user would:

  • dagger sdk install ./<sdk> succeeds and records the as-sdk marker
  • dagger module init <sdk> test-mod scaffolds the module, writes dagger-module.toml, installs it in dagger.toml, and records the authoring-SDK entry
  • dagger generate succeeds on the fresh scaffold, and is anchored at the caller's cwd (generate from inside one module's directory leaves a sibling module untouched)
  • the scaffolded module loads and serves its API (dagger api functions — templates may expose no functions yet)
  • dagger sdk module-options introspects initModule
  • dagger module engine required / dagger module deps list work from the module directory

Each lifecycle stage and each contract behavior is its own check; when a stage fails, dependent checks report a prerequisite command failed message naming that stage instead of repeating raw errors.

Carried forward from the old root module

  • initModule + init-template/ + helpers/render-init-template/ — sdk-sdk is still the SDK for authoring new Dang SDK helper modules (dagger module init sdk-sdk my-sdk)
  • The function-level contract checks (changeset hygiene), aligned with how the engine drives SDKs: explicit --path, @generate hook detected via dagger generate -l

Self-hosting instead of fixtures

sdk-sdk fulfills its own SDK contract (targetRuntime, initModule, a no-op @generate), so dagger check in this repo runs every check against sdk-sdk itself.

Housekeeping

  • Root config is dagger-module.toml (engineVersion v1.0.0-0) plus a dagger.toml workspace config
  • ./polyfill deleted in favor of github.com/dagger/polyfill; mod-test cut over to dagger api call -j

Verification (released v1.0.0-beta.9 CLI + engine)

  • In-repo dagger check: 22/22
  • go-sdk main: 19/19
  • dang-sdk main: 19/19
  • typescript-sdk main: 19/19
  • python-sdk main: 19/19

🤖 Generated with Claude Code

Add ./sdk-test, a CLI 1.0 module that black-box tests SDK modules
(go-sdk, dang-sdk, typescript-sdk, python-sdk) through a release
Dagger CLI, the way a user drives them:

  dagger -m github.com/dagger/sdk-sdk/sdk-test -W <sdk-repo> check

The checks vendor the SDK into a scratch git workspace inside a runner
container and exercise the full lifecycle: `dagger sdk install` marks
the SDK as-sdk, `dagger module init` scaffolds and registers a module
and records the authoring SDK, `dagger generate` succeeds, the
generated module serves functions, `dagger sdk module-options`
introspects initModule, and `dagger module engine`/`deps` verbs work
from the module directory.

When run inside this repository the checks redirect to a minimal
fixture SDK under .dagger/modules/sdk-test-e2e, keeping the harness
self-testing. Verified green against the fixture and dang-sdk on a
v1.0.0-beta.7 CLI/engine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: kpenfound <kyle@dagger.io>
@kpenfound
kpenfound requested a review from TomChv July 27, 2026 20:34
kpenfound and others added 2 commits July 28, 2026 13:18
The root sdk-sdk module now hosts the black-box lifecycle checks that
previously lived in ./sdk-test, alongside the function-level contract
checks and the Dang SDK helper scaffolding carried forward from the
previous root module:

  dagger -m github.com/dagger/sdk-sdk -W <sdk-repo> check

sdk-sdk now fulfills the CLI 1.0 SDK contract itself (targetRuntime,
initModule, a @generate hook), so `dagger check` in this repository
exercises every check against sdk-sdk as the SDK under test — no
fixtures needed. The contract checks were aligned with how the engine
actually drives SDKs: initModule is always called with an explicit
--path, and the @generate hook is detected via `dagger generate -l`
rather than by function name.

The root config is now dagger-module.toml (engineVersion v1.0.0-0)
with a dagger.toml workspace config, ./polyfill is replaced by the
canonical github.com/dagger/polyfill, and the polyfill-e2e and fixture
modules are removed. Verified green: 21/21 checks in-repo and 18/18
against a dang-sdk clone on a v1.0.0-beta.7 CLI/engine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: kpenfound <kyle@dagger.io>
@kpenfound kpenfound changed the title feat: add sdk-test module for black-box SDK contract checks feat!: replace root module with the black-box SDK test harness Jul 28, 2026
Run every lifecycle command (sdk install, module init, generate, and
the per-check commands) through a single capture wrapper. When a
command fails, later commands in the pipeline are skipped and record a
marker; their checks then fail with a "prerequisite command failed"
message naming the broken stage and its stderr, while the stage's own
check reports the actual error. The check report now shows exactly
which stage broke and which behaviors were blocked by it, instead of
repeating raw nested exec errors.

Stage states and their checks now share the same execs, so the
pipeline no longer runs install/init twice per pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: kpenfound <kyle@dagger.io>
@TomChv

TomChv commented Jul 29, 2026

Copy link
Copy Markdown
Member

We should add a part to test client generation, but that should only enabled through a setting because some SDK doesn't support client gen yet.
I think that could be done in a follow up though

Bump the pinned CLI release default to 1.0.0-beta.8 in both sdk-sdk
and mod-test ahead of the beta.8 release, and cut mod-test over to the
CLI 1.0 command surface: `dagger api call -j` replaces `dagger call
-j`, which was removed in beta.7. mod-test was the last piece pinned
to a pre-beta.7 CLI for the old verb.

The full suite (21/21 checks, including mod-test-e2e and the contract
checks that route through mod-test) passes with the new command
surface on the beta.7 CLI; runs will fail on the CLI download until
beta.8 artifacts are published, at which point no further change is
expected beyond re-validating the official SDKs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: kpenfound <kyle@dagger.io>
@TomChv

TomChv commented Jul 30, 2026

Copy link
Copy Markdown
Member

Do you think you could take care of dagger/dagger#13688 as part of the changes since you remove a lot of the modules? Do we even need that now?

@kpenfound

kpenfound commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

New list of functions in the sdk-sdk:

Name                                                 Type       Description
sdk-sdk:deps-list-succeeds                           check      `dagger module deps list` should succeed for a scaffolded module.
sdk-sdk:engine-required-reports-version              check      `dagger module engine required` should report a version for a scaffolded module.
sdk-sdk:generate-exposes-generator                   check      An SDK module should expose a `@generate` hook.
sdk-sdk:generate-respects-cwd                        check      `dagger generate` should be anchored at the caller's cwd.
sdk-sdk:generate-succeeds                            check      `dagger generate` should succeed on a freshly scaffolded module.
sdk-sdk:init-module-does-not-remove-existing-files   check      `initModule` should not remove existing files when creating a new module.
sdk-sdk:init-module-does-not-write-config            check      `initModule` should not write engine-owned config files.
sdk-sdk:init-module-honors-custom-path               check      `initModule --path` should stage files under the requested path.
sdk-sdk:init-module-renders-root-type                check      sdk-sdk's own `initModule` should render CLI module names as a valid Dang root type.
sdk-sdk:init-module-seeds-files                      check      `initModule` should seed SDK files for a new module.
sdk-sdk:init-records-authoring-sdk                   check      `dagger module init` should record the SDK as the new module's authoring SDK.
sdk-sdk:init-registers-module                        check      `dagger module init` should install the new module in dagger.toml.
sdk-sdk:init-scaffolds-module                        check      `dagger module init <sdk> <name>` should scaffold a new module.
sdk-sdk:init-writes-module-config                    check      `dagger module init` should write the new module's dagger-module.toml.
sdk-sdk:install-marks-as-sdk                         check      `dagger sdk install` should mark the SDK with an as-sdk marker in dagger.toml.
sdk-sdk:install-registers-sdk                        check      `dagger sdk install` should accept the SDK module.
sdk-sdk:scaffolded-module-serves-functions           check      A scaffolded module should serve at least one function after `dagger generate`.
sdk-sdk:sdk-reports-module-options                   check      `dagger sdk module-options` should introspect the SDK's initModule capability.
sdk-sdk:generate                                     generate   Regenerate the modules this SDK manages.

Add a generate-respects-cwd check encoding the cwd contract from
go-sdk's respect_cwd work: generation from inside one module's
directory acts on the module cone at or below that directory, so a
sibling module elsewhere in the workspace must be left untouched.

The harness scaffolds a second module (sibling-mod), commits the
scratch workspace as a git baseline, runs `dagger generate` from
inside the first module's directory, and asserts via git status that
the sibling saw no changes. For SDKs with no-op generation the check
passes vacuously; for codegen SDKs it catches cwd-ignoring generate
implementations. SdkRun gains workspaceChanges(path) for git-based
drift inspection.

Verified on a dev main engine (beta.8 preview): 22/22 in-repo, 19/19
against dang-sdk, and 17/18 against go-sdk main — the one failure is a
genuine go-sdk finding (a scaffolded module serves no functions after
init + generate).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: kpenfound <kyle@dagger.io>
…scaffolds

Bump the pinned CLI default from the unreleased 1.0.0-beta.8 to the
released 1.0.0-beta.9, and relax scaffolded-module-serves-functions
into scaffolded-module-loads: `dagger api functions` must succeed
against a fresh scaffold, but starter templates may expose no
functions — go-sdk deliberately scaffolds an empty root type, and
"No functions found." on a loadable module satisfies the contract.
A broken scaffold (e.g. missing generated bindings) still fails the
check at module load.

Validated on the released beta.9 CLI and engine: 22/22 in-repo, and
19/19 against the latest main of go-sdk, dang-sdk, typescript-sdk,
and python-sdk.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: kpenfound <kyle@dagger.io>
@kpenfound
kpenfound marked this pull request as ready for review August 4, 2026 17:58
@TomChv

TomChv commented Aug 5, 2026

Copy link
Copy Markdown
Member

Tested on the typescript-sdk repo and it works as expected:

TRACE  ✔ PASSED
dagger-1.0.0-beta.9 -m github.com/dagger/sdk-sdk@sdk-test -W . check  1m9s

CHECKS  ✔ 19 passed
✔ sdk-sdk:deps-list-succeeds 1m3s OK
✔ sdk-sdk:engine-required-reports-version 1m3s OK
✔ sdk-sdk:generate 1.1s OK
✔ sdk-sdk:generate-exposes-generator 17.4s OK
✔ sdk-sdk:generate-respects-cwd 1m2s OK
✔ sdk-sdk:generate-succeeds 1m2s OK
✔ sdk-sdk:init-module-does-not-remove-existing-files 29.1s OK
✔ sdk-sdk:init-module-does-not-write-config 29.3s OK
✔ sdk-sdk:init-module-honors-custom-path 29.3s OK
✔ sdk-sdk:init-module-renders-root-type 16.0s OK
✔ sdk-sdk:init-module-seeds-files 29.1s OK
✔ sdk-sdk:init-records-authoring-sdk 29.5s OK
✔ sdk-sdk:init-registers-module 29.5s OK
✔ sdk-sdk:init-scaffolds-module 29.4s OK
✔ sdk-sdk:init-writes-module-config 29.5s OK
✔ sdk-sdk:install-marks-as-sdk 15.5s OK
✔ sdk-sdk:install-registers-sdk 15.5s OK
✔ sdk-sdk:scaffolded-module-loads 1m5s OK
✔ sdk-sdk:sdk-reports-module-options 17.4s OK

@TomChv
TomChv merged commit 8c16442 into main Aug 5, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants