acceptance: schema-driven invariant fuzzing for bundle configs - #5686
Open
radakam wants to merge 70 commits into
Open
acceptance: schema-driven invariant fuzzing for bundle configs#5686radakam wants to merge 70 commits into
radakam wants to merge 70 commits into
Conversation
Collaborator
Integration test reportCommit: af4172f
9 interesting tests: 4 RECOVERED, 4 SKIP, 1 flaky
Top 38 slowest tests (at least 2 minutes):
|
Contributor
Approval status: pending
|
denik
reviewed
Jun 26, 2026
…n config The parent test.toml sets GOOSOnPR.windows/darwin=false and adds job_run.yml.tmpl, but the per-directory out.test.toml snapshots were stale, failing the CI "detected changed or new files" check.
Four improvements to the bundle schema fuzzer: - Oracle: in fuzz mode (SKIP_DRIFT_CHECK) no_drift/redeploy now assert plan determinism (two consecutive `plan` reads must be byte-identical) instead of only no-panic. This is independent of fake-server fidelity, unlike no-drift or no-destructive-drift (an unchanged config's recreate is just a local-vs-remote representation mismatch on an immutable field). redeploy also panic-scans its second deploy while tolerating fidelity-driven redeploy failures. - Coverage: add `task test-fuzz-cover` to run the corpus under -cover and report per-package coverage (0.0% = never exercised), plus opt-in FUZZ_CORPUS_DIR to persist deployable configs as a generator-independent regression corpus. - Rejection waste: emit non-ASCII scalars as literal UTF-8 (ensure_ascii=False) so hostile values reach bundle logic instead of dying at the YAML parser as an invalid surrogate-pair escape (was the single largest rejection bucket). - Truncation: FUZZ_TIME_BUDGET now defaults on (900s, under the 20m Timeout) so a slow variant stops cleanly and passes rather than being force-killed into a false failure; FUZZ_TIME_BUDGET=0 restores an uncapped run.
Add a schema-aware additive op to mutate_fuzz_config: alongside deleting and perturbing existing fields, it now injects a valid optional field the curated base omits, valued by the schema generator. Destructive mutations stay within the base's field set (only reject/panic bugs); adding a valid optional field to a still-deploying config is what reaches reconcile/drift bugs. The no-schema path is unchanged (RNG stream and selftest golden preserved).
… fix, gitignore - Remove the orphaned edit_fuzz_config.py chain (edit script, its check, and selftest); the update invariant it served was removed earlier in this branch. Decouple gen_fuzz_config_check.py from it (the one-line scalar contract is now attributed to mutate_fuzz_config's loader, which actually relies on it). - Revert the libs/testserver/catalogs.go create-payload round-trip; it is a standalone fake-server fidelity change unrelated to the fuzz harness. - Drop the .gitignore /build/ and .fuzztmp/ entries (stale/misleading paths). - Regenerate fuzz/out.test.toml (drops stale GOOSOnPR lines).
no_drift's post-deploy plan already dry-runs what a redeploy would apply, so
it catches the same field-level non-idempotency without the cost of a second
deploy; the only surface redeploy adds (plan/apply divergence) never surfaced
a bug across the nightly runs. Multi-resource generation only added
cross-resource reference injection, which likewise found no distinct defect
(every bug reproduces at a single resource) while lowering acceptance.
Remove both, shrinking the fuzz matrix from 12 leaf variants to 4
({no_drift, migrate} x {generate, mutate}). Regenerate out.test.toml and
update the README accordingly.
# Conflicts: # acceptance/bundle/invariant/no_drift/script
…inism diff The deploy_rc capture in no_drift/script and migrate/script was dead under `bash -euo pipefail`: a failing deploy aborted at the `trace` line before the capture, so the panic check never ran on a failed deploy. Wrap the deploy in `set +e`/`set -e` so the code is captured, the panic check runs even on failure (a panicking-but-rejected config is a bug, not a rejection), and only then do we exit with the deploy's code. Gate the pre-deploy `bundle validate` on FUZZ_SEED: deploy runs the same validate pipeline, so it is redundant for curated configs and only added value as an isolated panic surface for fuzzed configs. This also restores the curated tests to their prior behavior. Guard the fuzz-mode plan-determinism oracle to diff only when both plans succeed: a plan that fails on an unstubbed read is a coverage gap, and its partial output can differ run-to-run, which would false-positive. Reword the load_plan comment to drop the inaccurate fuzzer reference (only the curated drift check reaches it).
Rename fuzz_gen_config.py to emit_fuzz_config.py so it no longer collides with the schema-walk generator gen_fuzz_config.py. Extract the shared config-render, cleanup, and deploy-capture prologue out of the no_drift and migrate scripts into prologue.sh. Shorten the comments added in this PR.
The rename to emit_fuzz_config sorts before envsubst, so ruff I001 flagged the import block in mutate_fuzz_config_check.py.
The invariant suite runs on cloud, and there each seed's deploy/migrate/plan/ destroy round trip against a real workspace takes minutes, tripping the script's 180s per-seed SEED_TIMEOUT and reporting a false hang. The cloud run also adds no signal: drift checking is off unless FUZZ_CHECK_DRIFT is set, which it does not set, so it only re-asserts the no-panic property the local run already covers over the same seeds in seconds.
mutate mode rendered the curated base template with the harness-wide UNIQUE_NAME, so every seed in a run deployed the same bundle root and the same securable names into one shared fake workspace. Anything a seed left behind then read back as remote state for a later seed: the testserver never drops grants on delete, so seeds 34 and 51 of no_drift/mutate failed on a grant an earlier seed had created and destroyed. generate mode already scopes names per seed; do the same here. no_drift/ mutate now runs to seed 71, the real allow_duplicate_names bug.
…gets no_drift, migrate, continue_293, delete_idempotent and destroy_idempotent each carried their own copy of the config render (data copy, init script, envsubst), the destroy-on-exit trap, and the deploy + panic-scan + INPUT_CONFIG_OK sequence. Move all of it into acceptance/bundle/invariant/script.prepare, which the harness picks up for the whole subtree, so each script holds only the invariant it asserts. The helpers are functions rather than prepare-time side effects because the harness concatenates script.prepare outside the subshell it wraps the script in, so a trap installed at prepare time would belong to the outer shell. invariant_deploy takes the deploy log name as its first argument, since the targets log to LOG.deploy, LOG.deploy_initial and LOG.deploy.293. Incidental deltas, all confined to LOG* files (logged, not compared): continue_293 gains LOG.config and a traced cleanup destroy, and destroy_idempotent's trap logs to LOG.destroy rather than LOG.destroy_final.
Drop the comments that restated the line below them and condense the helper docs, keeping only the non-obvious parts: why invariant_setup must be called from the target rather than at prepare time, and what INPUT_CONFIG_OK signals to the fuzzer.
The fuzzer could pass while every seed was rejected, so a broken generator, schema or fixture looked identical to the CLI correctly rejecting random input. Assert that at least one seed deploys, and treat a generator crash as a bug rather than a rejected config. Also fixes classification and reporting the review turned up: - an unstubbed testserver route hit during plan or destroy was reported as a CLI drift bug; check for it before the INPUT_CONFIG_OK marker - repro hints were mangled by the harness env-var replacements; write them to LOG.repro, which is logged verbatim - cleanup skipped destroy unless deploy fully succeeded, leaking resources from a part-way deploy on a real workspace - to_yaml flattened nested lists ([[1,2]] -> [1,2]) and the loader could not read the empty containers it emits - gen_grants/gen_permissions emitted [] for types with no valid privilege, manufacturing the known empty-grants drift bugs instead of new ones - CatalogsCreate dropped connection_name, custom_max_retention_hours and managed_encryption_settings, so the next plan saw a phantom change - mutate + migrate only rediscovers what migrate/test.toml already excludes Drops the schema-type guard (gen_scalar already exits on an unknown type), the write-only FUZZ_CORPUS_DIR, the never-set FUZZ_RESOURCES, both unused argparse entry points and single-caller util.load_plan, whose inlining also restores the raw-plan dump on malformed JSON.
The name is the identity of both resources, so a fake that accepts an empty one stores a resource the CLI cannot find again: deploy reports success and then fails with "internal error: missing entry in state after deploy", and destroy fails with "internal error, missing in state". Verified against a real workspace that both APIs reject it with 400 INVALID_PARAMETER_VALUE, so the fuzz seeds that hit this were reporting a fake-server gap as a CLI bug. Also two harness fixes found while replaying seeds: - pass -count=1 to the fuzz tasks. Only the script reads FUZZ_*, so Go's test cache happily served one seed window's result as another's. - skip the no-seed-deployed assertion for a single-seed replay, where one rejected config is the expected outcome rather than systemic breakage.
Guard INPUT_CONFIG in invariant_cleanup: scripts run under set -u, and a caller that builds its own config leaves the variable unset. Split invariant_render out of invariant_setup so such a caller can override the render alone from its own script.prepare, which the harness concatenates after this file. Lift set -e around the deploy in invariant_deploy so the panic scan still runs when the deploy fails: a config the CLI rejects must not panic on the way out. A clean failure exits with the deploy's own code and prints no INPUT_CONFIG_OK.
no_drift, migrate and continue_293 each ran the same plan + panic-scan + verify_no_drift.py sequence. Move it into invariant_verify_no_drift so a caller whose config the server does not round-trip exactly can override the check alone. migrate now logs the plan to LOG.planjson like the other two.
#6075 replaces the sourced prologue.sh with acceptance/bundle/invariant/script.prepare, which the harness concatenates for the whole subtree, and turns the shared setup into helpers a child script.prepare can override. - Drop prologue.sh along with its FUZZ_SEED branch; no target script sources anything now. - Add fuzz/script.prepare, overriding invariant_render to emit the generated config and run the validate panic probe, and -- unless FUZZ_CHECK_DRIFT is set -- invariant_verify_no_drift with the plan-determinism oracle. - Drop SKIP_DRIFT_CHECK: no_drift and migrate no longer branch on fuzz mode. migrate now runs the determinism oracle in fuzz mode rather than skipping the drift check entirely. - Guard INPUT_CONFIG in migrate/script, which the fuzzer leaves unset.
The fuzzer clears the INPUT_CONFIG matrix, and scripts run under set -u, so the depends_on special case must not dereference it unguarded.
delete_idempotent and destroy_idempotent assert that a delete or destroy re-run against state that still references the removed resources succeeds. That holds regardless of how faithfully the fake server round-trips fields, so unlike no_drift and migrate these targets keep their real oracle under fuzzing, and they exercise the already-gone paths across generated resource types. Both derive workspace paths from test-bundle-$UNIQUE_NAME, which the fuzzer has to render the config with. Scope the per-seed unique name in the seed loop rather than inside emit_fuzz_config.py, where the rewrite never reached the shell, and name generated bundles after the same pattern as the curated configs. 200 seeds per target per mode run clean locally: 143/200 deploy in generate mode (14 testserver gaps) and 169/200 in mutate mode, with no bugs or hangs.
Generate mode adds no axis the curated delete_idempotent and destroy_idempotent tests lack. The already-gone delete path is selected by resource type, and those tests already run every type in the INPUT_CONFIG matrix; generate pins name and display_name, so it cannot vary the identifier the delete call uses, and its dangerous values only reach free-form strings no delete path reads. Mutate can land a dangerous value on the identifier and can reshape grants and permissions, so it keeps both targets. Leaves 5 leaf variants and drops the testserver coverage gaps generate produced for these two.
# Conflicts: # acceptance/bundle/invariant/script.prepare #6075 landed as a pure extraction, so main's script.prepare no longer carries the two deltas this branch had added to it. Keep main's version and re-add only the INPUT_CONFIG guard in invariant_cleanup, which this branch does need: the fuzzer clears the INPUT_CONFIG matrix and scripts run under set -u. The set -e lift around the deploy is not re-added. It is redundant here -- when a seed fails, fuzz/script already scans every LOG.* for panics before classifying it, so a config that panics on the way out is still reported as a bug rather than a rejection.
The seed loop wraps run_seed in `set +e` to capture the seed's exit code. The capped branch spawns a fresh `bash -euo pipefail`, but the uncapped fallback is a plain subshell that inherits `set +e`, so the seed ran without errexit: a config the CLI rejects continued past the failing validate and deploy to `echo INPUT_CONFIG_OK`, and that marker is what tells "deployed, then broke the invariant" from "config rejected". Every seed the CLI legitimately rejected was reported as a bug. macOS has no GNU timeout, so it always takes the fallback and was the only platform failing; FUZZ_SEED_TIMEOUT=0 selects the same branch and reproduces it anywhere. Seed 3 of generate mode (pipelines.*.root_path pointing at a path that does not exist) failed as a bug with the flag and passes without it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an acceptance-test fuzzing harness that stresses the bundle engine by generating random configs and asserting deploy invariants. Since the schema comes from the CLI under test, a failure is a real bug: panic, internal error, or drift.
What it does
acceptance/bin/gen_fuzz_config.py) builds a random resource by walking the livedatabricks bundle schema. Reference and typed fields (catalog, schema,parent_path, file/notebook paths, durations) are pinned to deployable values, and ~15% of free-form scalars get a dangerous / near-range-end value: empty, whitespace, over-long, control chars, non-ASCII, path traversal, int32/int64 boundaries.acceptance/bin/mutate_fuzz_config.py) starts from a curated invariant config and applies 1-3 seeded mutations: destructive (delete a field, set a fuzz token, set a dangerous value, empty a container) and, with the schema, additive (inject a valid optional field the base omits).acceptance/bin/emit_fuzz_config.pydispatches onFUZZ_MODE.acceptance/bundle/invariant/fuzz/script.prepareoverrides two helpers, so fuzzing stays invisible to the targets:invariant_renderemits the generated config and probesbundle validatefor panics before the deploy.invariant_verify_no_driftasserts the fidelity-independent property: planning is deterministic, so two consecutive plans of the same state are byte-identical.FUZZ_CHECK_DRIFTselects the exact drift check instead, which the nightly job sets.acceptance/bundle/invariant/fuzz/script) classifies each seed as deployed / rejected / gap / bug / hang, so genuine bugs stand apart from config rejections and testserver coverage gaps. It caps each seed (FUZZ_SEED_TIMEOUT, SIGQUIT→SIGKILL so a hang dumps goroutines) where GNUtimeoutis available and runs the seed uncapped where it is not; both paths seterrexitexplicitly, so a rejected config stops before theINPUT_CONFIG_OKmarker and classification does not depend on the platform. It bounds the whole variant with clean truncation (FUZZ_TIME_BUDGET), and tallies seeds per variant inLOG.summary. A run where no seed deploys fails: that means the schema, generator or fixtures are broken, which otherwise looks identical to the CLI correctly rejecting random input.Targets and modes. 4 targets over 2 modes, less 3 excluded combinations, gives 5 leaf variants:
The idempotency targets keep a real oracle under fuzzing — a delete or destroy re-run against state that still references the removed resources must succeed, however faithfully the fake server round-trips fields. Mutate mode is the one that varies what they depend on: the identifier the delete call uses, and the shape of grants and permissions.
Where it runs. Locally against the test server, where a seed's full deploy/plan/destroy round trip takes seconds, so a wide seed window fits in one job. The committed run is 5 seeds; the nightly
task test-fuzzjob walks a rotating window with the exact drift check on.Supporting changes
Taskfile.yml:task test-fuzz(wide seed window, drift check on) andtask test-fuzz-cover(runs the fuzzer under-coverwith drift off and reports per-package CLI coverage, so a package at 0.0% is one the fuzzer never reaches)..github/workflows/push.yml: nightly cron job over a rotating, non-overlapping seed window (fromGITHUB_RUN_NUMBER) that comments failures on the PR that introduced the commit, with a single-seed repro.libs/testserver:IgnoreUnhandledRequests, atest.tomlfield plumbed throughstartLocalServer, returns 501 for an unmodeled route so the runner records the seed as a coverage gap. Scoped to the fuzz test, so missing-handler bugs stay loud everywhere else.libs/testserverfidelity, both needed to stop the fake server reporting its own gaps as CLI bugs:CatalogsCreateechoes backconnection_name,custom_max_retention_hoursandmanaged_encryption_settingsinstead of dropping them into a phantom change on the next plan, and catalogs and registered models reject an emptynamewith a 400 as the backend does — accepting one stored a resource the CLI could not find again, surfacing asinternal error: missing entry in state after deploy.gen_fuzz_config_check.pyandmutate_fuzz_config_check.pypin the generators' contracts (the one-line scalar shape the mutate loader relies on, loader round-trip over every curated base, mutation determinism, no output-only field injection) via acceptance selftests, andverify_no_drift.pyreports empty plan output as such instead of raising.Reproducing a failure
FUZZ_SEED_TIMEOUT=0selects the uncapped seed path, which is also what a platform without GNUtimeout(macOS) takes.Bugs found by the fuzzer