bundle: record and read deployment state via DMS by shreyas-goenka · Pull Request #6074 · databricks/cli

shreyas-goenka · 2026-07-27T00:45:01Z

Records bundle deployment state with the deployment metadata service (DMS) and reads it back, so DMS — not the local state file — becomes the source of truth for what a bundle has deployed.

Supersedes #6052, which this PR contains; review here instead. Restricted to net-new deployments: see Only for net-new deployments.

What it does

Direct-engine deploys record their state with DMS under /api/2.0/bundle, and read it back on the next deploy.

Write path. A deploy opens a version, records one operation per applied resource, and completes the version:

CreateDeployment mints a deployment the first time a bundle records anything. The ID is server-generated and persisted in the local state header, so subsequent deploys reuse it. Destroy calls DeleteDeployment.
CreateVersion opens a version with optimistic concurrency (version_id == last_version_id + 1; a concurrent deploy gets 409 ABORTED). A heartbeat holds the lease while the deploy runs.
CreateOperation records each applied resource: its action, its ID, and the applied config. Deletes record no state.
CompleteVersion closes the version SUCCESS or FAILURE.

Read path. A deployment whose last_successful_version_id is set is DMS-owned: the next deploy fetches its resources and overlays them onto the local state, keeping only the local identity (lineage, serial, deployment ID) and the locally-tracked dependency edges, which DMS does not record.

Operations upload asynchronously. CreateOperation does not block the resource being deployed:

record serializes the operation and pushes the resource key onto a buffered channel (10 slots — deeper than defaultParallelism is wide, so an apply worker practically never blocks on a free slot).
A pool of 4 workers drains it. The queue, the pool, and the coalescing are one type, operationQueue (bundle/direct/opqueue.go).
If an operation for a resource is still waiting when a second is recorded for it, the waiting one is replaced rather than queued again. DMS keeps one state per resource key, so the later operation supersedes the earlier one and a single request records both. Best effort: only operations nobody has picked up yet collapse.

Two subtleties there:

At most one upload in flight per resource key. An inflight set stops a coalesced key from being handed to a second worker while the first still owns it; the owner re-checks pending after each upload instead of re-queueing (a worker sending to the channel it consumes from deadlocks once the queue is full). Otherwise two uploads for one key could land out of order and leave a stale state behind. TestOperationQueueUploadsOneResourceAtATime pins it.
Not fire-and-forget. Apply drains the queue after g.Run returns — every record call has returned by then — and reports the first upload error. A version that completes successfully makes DMS authoritative, so a silently dropped operation would leave DMS with an incomplete resource set and the next deploy would plan to create resources that already exist. The drain also has to happen inside Apply, before the deferred CompleteVersion.

Terraform deploys are untouched: DMS state is tracked per direct-engine deployment, and only the direct engine opens the state DB that holds the deployment ID.

Only for net-new deployments

experimental.record_deployment_history is only honored for a bundle with no deployed resources yet. If the state file already tracks resources that DMS does not know about, the deploy fails with an actionable error instead of recording:

Error: cannot record deployment history for a bundle that already has deployed resources tracked in
.databricks/bundle/default/resources.json: only new deployments can be recorded. Remove
experimental.record_deployment_history, or destroy the bundle and deploy it again

Once DMS holds a successful version it is authoritative for resource state even when its resource set is empty. Adopting a state file written by a CLI that predates DMS would therefore make already-owned resources look absent, and the next deploy would create them a second time.

Two cases are allowed through: a state file DMS already owns (it carries a deployment ID — a bundle that opted in while it was still empty), and a state file with no resources. The latter matters because destroy leaves resources.json on disk with an empty resource set, which is what makes the error's destroy-and-redeploy advice actually work. The check keys off len(State) rather than the file existing for exactly that reason.

Upgrading an existing state in place is deliberately left out of this PR and marked TODO(DMS): it means writing the state at featureStateVersion (3) with a feature flag recording that DMS owns it, plus a tombstone entry per resource so a CLI that predates DMS refuses the state rather than silently deploying against a resource set it cannot see. The forward-compat scaffolding for that (featureStateVersion, Header.Features) already exists.

Notes for reviewers

A few things that are non-obvious and are commented in the code:

last_successful_version_id is stage: DEVELOPMENT and is stripped from the generated SDK, so it is read with a temporary raw GET marked TODO(DMS).
The testserver's GetDeployment serializes through a map, not a struct embedding bundledeployments.Deployment. That type has its own MarshalJSON, which Go promotes to the embedding struct and which drops the sibling last_successful_version_id field — silently disabling the entire read path in tests.
A deploy that applies nothing still persists a deployment-ID change (headerDirty), or the server-assigned ID would be thrown away and the next deploy would create a second deployment record.
A missing deployment on read means "create a new one" — a destroy deletes the record but leaves its ID in local state, so a fatal 404 there would make every subsequent deploy fail unrecoverably. Any other read error stays fatal, so a transient failure cannot silently fork a second deployment.
Recorded state goes through structwalk.RedactSensitiveFields, matching what dstate.SaveState does, so a bundle:"sensitive" field cannot reach DMS in plaintext (and cannot come back into a local state file through the read path). Latent today — no resource state type carries one yet.

Tests

Acceptance tests, all direct-engine:

bundle/dms/record — the full write path, including the recorded state payload.
bundle/dms/multiple-resources — five resources, exactly one operation each regardless of upload order; no operations on a no-op redeploy.
bundle/dms/redeploy-after-destroy — deploy → destroy → deploy, showing the 404s against the destroyed ID followed by a fresh POST /deployments.
bundle/dms/no-resources — one deployment across two deploys, with the ID round-tripping through state.
bundle/dms/existing-state — deploying without recording, then enabling it: the error, and that no deployment reaches DMS. Then destroy → enable → deploy, showing the advice in the error message works and the redeploy records normally.

Unit tests cover the recorder and serialization, the queue (per-operation upload, coalescing while an upload is held, error propagation through the drain, serialization failure at record time, idempotent close, one-upload-at-a-time under 10 concurrent producers over 12 keys, nil queue), the testserver response shape, stale-vs-other GetDeployment errors, deployment-ID persistence with no resource entries, depends_on preservation across the overlay, and sensitive-field redaction.

The direct-engine acceptance failures I see locally reproduce identically on the base commit — sandbox environment issues, not regressions.

This pull request and its description were written by Isaac.

…d IDs Wire the direct engine into the Deployment Metadata Service (DMS) so that a `record_deployment_history`-enabled bundle records each deploy/destroy as a version and can read its resource state back from DMS. The deployment ID is now assigned by the server: the first deploy calls CreateDeployment with an empty ID, reads the assigned ID back from the response, and persists it in the direct-engine state header (Header.DeploymentID). Later deploys pass the stored ID back, so a bundle maps one-to-one to a DMS deployment even after the local cache is deleted (the ID rides along in the workspace-synced state file). - libs/dms: Recorder creates the deployment (server-assigned ID) + version, heartbeats the lease, completes it, and deletes the deployment on destroy. - bundle/direct: operationRecorder reports each applied resource operation; the wire resource_key drops the CLI-internal "resources." prefix. - bundle/direct/dstate: Open takes a DMS client and overlays DMS resource state when DMS holds a successful version; deployment ID persisted in the header. - bundle/phases: create the version after plan approval, complete it under the lock, record operations during apply. - libs/testserver: stateful fake DMS (deployments/versions/operations/resources) with server-generated IDs; acceptance test covers deploy, cache-loss redeploy, and destroy. Co-authored-by: Isaac

The read overlay decided whether DMS owns a deployment's state by listing versions and scanning for a successful one. The deployment now exposes last_successful_version_id directly, so a single GetDeployment answers the same question — no version listing. The field is still stage:DEVELOPMENT in the proto and therefore stripped from the generated SDK, so this reads the deployment via a raw GET into a local struct as a temporary stub. Once the field is promoted to PRIVATE_PREVIEW and regenerated, the raw call collapses to client.GetDeployment(...). LastSuccessfulVersionId and the threaded config argument goes away (see the TODO in deploymentHasSuccessfulVersion). The testserver's GetDeployment now serves last_successful_version_id (tracked on version completion), and the now-unused ListVersions fake is removed. Co-authored-by: Isaac

Five fixes to the DMS state recording added in databricks#6052: 1. The fake DMS server dropped last_successful_version_id. GetDeployment serialized the response through a struct embedding bundledeployments.Deployment, whose promoted MarshalJSON silently discards sibling fields. The CLI reads a missing value as "DMS does not own the state", so the entire read/overlay path (overlayDMSState, fetchDeploymentResources, deploymentHasSuccessfulVersion) never ran in any test. Serialize through a map instead, and unit-test the shape. 2. A bundle with no resources leaked a deployment record per deploy. Such a deploy writes no WAL entries, and the state file was only persisted when the WAL carried entries, so the server-assigned deployment ID was dropped and the next deploy created a second deployment. Track a dirty header so Finalize persists an ID change on its own. A header-only WAL that changed nothing still skips the write, keeping the serial in step (acceptance/bundle/deploy/wal/header-only-wal). 3. Deploy after destroy failed permanently. A successful destroy deletes the deployment record but leaves its ID in local state, so the next deploy's GetDeployment 404'd and the error was fatal — unrecoverable on retry. Treat a missing deployment as "create a new one"; any other read error stays fatal. 4. The overlay dropped depends_on. DMS does not record dependency edges, so replacing local state with DMS resources lost them, affecting delete ordering, the apply graph, and --select expansion. Carry depends_on over from the local entry. Masked by (1) until now. 5. Recording bypassed secret redaction. dstate.SaveState redacts bundle:"sensitive" fields before writing state, but the operation recorder marshalled raw, so a secret would be sent to DMS in plaintext and read back into local state. Route through structwalk.RedactSensitiveFields. Latent today: no resource state type carries a sensitive field yet. Adds acceptance coverage for deploy-destroy-deploy and for a bundle with no resources, both of which now exercise the read path (visible as GET .../resources in the recorded requests). Co-authored-by: Isaac

Recording an operation with the deployment metadata service used to happen inline on the apply worker, so every resource paid a CreateOperation round trip before the worker moved on to the next one. Queue the operations instead and upload them from a small pool of background workers (operationQueue, in the new opqueue.go). The queue holds resource keys rather than operations, so an operation recorded for a resource that is still waiting replaces the queued one: DMS keeps one state per resource key, so the later operation supersedes the earlier one and a single request records both. This is best effort - only operations that no worker has picked up yet are coalesced. Uploads are not fire-and-forget. Apply drains the queue before returning and reports the first failure, because a version that completes successfully makes DMS authoritative for resource state; dropping an operation would leave DMS with an incomplete resource set and the next deploy would plan to create resources that already exist. At most one upload per resource key runs at a time, so the last operation recorded for a resource is also the last one the service sees. Co-authored-by: Isaac

Recording deployment history is implemented end to end, but it cannot be exposed to users yet: enabling it makes the deployment metadata service the source of truth for resource state, and there is no upgrade path from an existing direct-engine state file to a DMS-owned one. Setting the flag is now an error. DATABRICKS_BUNDLE_ENABLE_RECORD_DEPLOYMENT_HISTORY lifts the error so the CLI's own acceptance tests and DMS development can exercise the feature until the direct state upgrade lands. Co-authored-by: Isaac

DATABRICKS_BUNDLE_ENABLE_RECORD_DEPLOYMENT_HISTORY read as if it were the switch that turns the feature on. It is not: the flag in databricks.yml does that, and this variable only permits the flag to be set while the feature is gated off. Name it DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY. Co-authored-by: Isaac

Replace the blanket gate on experimental.record_deployment_history with a narrower check in dstate.DeploymentState.Open: recording is refused only when the state file already tracks deployed resources that DMS does not know about. Once DMS holds a successful version it is authoritative for resource state even when its resource set is empty, so adopting a state file written by a CLI that predates DMS would make already-owned resources look absent and create them a second time. A state file that DMS already owns is fine, and so is one with no resources (e.g. left behind by a destroy), which is what makes the error's destroy-and-redeploy advice work. The check keys off len(State) rather than the file existing because destroy leaves resources.json in place with an empty resource set. This drops validate.ValidateRecordDeploymentHistory and the DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY escape hatch, which are no longer needed. Upgrading an existing state in place (state v3 with a feature flag plus per-resource tombstones so older clients refuse the state) is left as a TODO. Co-authored-by: Isaac

The concurrent-producer test tripped three linters: - modernize/revive want wg.Go instead of wg.Add + go func + defer wg.Done. - testifylint's go-require flags recordState, which calls require inside the spawned goroutines. testify assertions may only run on the goroutine running the test function. Record inline and send each error to a buffered channel that the test goroutine drains after wg.Wait, so the assertions stay on the test goroutine. Co-authored-by: Isaac

shreyas-goenka · 2026-07-27T16:11:06Z

+	// operationQueueSize bounds how many recorded operations wait for upload.
+	// Apply deploys at most defaultParallelism resources at a time, so a queue
+	// this deep means an apply worker practically never blocks on a free slot.
+	operationQueueSize = 10


tunable - we can increase the queue size and workers with some benchmarks in the future.

The operation queue collapses repeated writes to the same resource key by replacing the queued operation wholesale, so a create followed by an update was uploaded as an update. That tells DMS the resource already existed before this deploy, when in fact this deploy created it. Merge the actions instead: the state uploaded is still the later one, but a queued create or recreate wins over a subsequent update. A delete still wins over anything queued before it, since the resource is gone. This is not reachable from Apply today - each resource is recorded once per deploy, because there is one record call per graph node and dagrun visits each node exactly once - so this is about the queue being correct for any caller that records a resource more than once. Co-authored-by: Isaac

Co-authored-by: Isaac

The deployment ID was persisted in the local state file header, which made the CLI the source of truth for a value the service mints. DMS registers each deployment as a workspace node named resources.deployment.json under initial_parent_path, and the node's ID *is* the deployment ID, so it can be resolved from the workspace instead. ResolveDeploymentID does a get-status on <state_path>/resources.deployment.json and returns the node ID, or empty when the node is absent. Deploy, destroy, and the read path all resolve the ID that way and pass it down; the read path then constructs state from GetDeployment + ListResources as before. Consequences: - Header.DeploymentID, GetDeploymentID, and SetDeploymentID are gone, along with the headerDirty machinery that only existed to persist the ID on a resource-less deploy. - Open takes a *DMSSource instead of a (client, config) pair, since the resolved ID now has to be threaded in too. - CreateDeployment sets initial_parent_path, which the service requires and the CLI never set. - createDeploymentVersion no longer recovers from a 404 on GetDeployment by creating a second deployment. A destroy trashes the node, so a resolved ID whose record is missing means the two are out of sync, and creating another deployment would collide on the same node path. The testserver models the real derivation: CreateDeployment creates the workspace node and uses its object ID as the deployment ID, so the acceptance tests exercise get-status resolution end to end. dms/record now wipes the local cache before redeploying and records zero operations, which is the read path reconstructing state entirely from DMS. Co-authored-by: Isaac

shreyas-goenka · 2026-07-27T23:44:35Z

+// and therefore stripped from the generated SDK. Once the field is promoted to
+// PRIVATE_PREVIEW and regenerated, replace the raw call with
+// client.GetDeployment(...).LastSuccessfulVersionId and drop DMSSource.Config.
+func deploymentHasSuccessfulVersion(ctx context.Context, cfg *sdkconfig.Config, deploymentID string) (bool, error) {


Can be simplified on the next SDK bump

- Limit recorded state to 64 KB, checked when the payload is built so an oversized resource fails itself rather than the drain at close. - Collapse the operation queue's pending/inflight pair into one `owned` set. The two maps encoded a single question ("is this key already claimed?") and had to be read together; `take` now only releases ownership. - Only log "Coalescing" when an operation was actually merged. The old code logged it for in-flight keys too, where nothing was coalesced. - Restore mergeWalIntoState's `hasEntries` naming and comment from main. The `persist` rename existed for the headerDirty case, which is gone. - Trim the comments added by this PR. Also note in fetchDeploymentResources that DMS has no field for dependency edges and they cannot be recovered from the recorded state (references are resolved to literals before serialization), so depends_on is carried over from the local state file. Co-authored-by: Isaac

The read path carried depends_on over from the local state file, which is empty exactly when it matters: a fresh checkout reconstructs state from DMS and got no dependency edges. Deletes are the one case that cannot recompute them, because the resource is gone from config, so two dropped resources with a real dependency could be deleted in the wrong order. DMS has no field for dependency edges, and they cannot be recovered from the recorded config either: references are resolved to literals before it is serialized. So Operation.State now carries an envelope, dstate.RecordedState, holding the config plus depends_on. Nesting depends_on inside the config would have collided with resource fields of the same name (jobs.Task.depends_on). The envelope mirrors the local ResourceEntry, so both sides of the round trip have the same shape. acceptance/bundle/dms/depends-on covers it: a job referencing another records its edge, and after wiping the local state a destroy still deletes the referencing job first. Co-authored-by: Isaac

shreyas-goenka · 2026-07-28T10:52:02Z

+//
+// The shape deliberately matches the local ResourceEntry so both sides of the
+// state round trip look the same.
+type RecordedState struct {


We can defer to resources.json for state version bumps. We can also inline that into DMS. Either option works well enough. WDYT @denik? It's not a one way door.

Migrating an existing deployment is not supported: Open rejects a bundle that already has resources in state, so a deployment that exists in DMS was created by an opted-in CLI and DMS owns its resource set outright. The last_successful_version_id probe that decided whether to trust DMS was therefore always true by the time it ran. Removing it takes with it the raw GET that read the field (it is stage:DEVELOPMENT and stripped from the generated SDK) and DMSSource.Config, which existed only to make that call. overlayDMSState is now readDMSState, since it no longer overlays anything conditionally: it just reads. Recording stays opt-in via experimental.record_deployment_history; that flag is what makes the caller pass a DMSSource at all. acceptance/bundle/dms/existing-state also now covers wiping the local cache: deploy pulls the state file back from the workspace, so the resources stay tracked and opting in is still rejected. Co-authored-by: Isaac

github-actions · 2026-07-28T11:03:51Z

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

PR number: 6074
Commit SHA: 386a0b616f6a409ab1c31fb34383b74bcd098547

Checks will be approved automatically on success.

shreyas-goenka added 5 commits July 24, 2026 01:41

shreyas-goenka changed the base branch from dms-state-recording to main July 27, 2026 13:58

shreyas-goenka changed the title ~~bundle: fix DMS deployment recording bugs found in review of #6052~~ bundle: record and read deployment state via DMS Jul 27, 2026

shreyas-goenka added 3 commits July 27, 2026 14:08

shreyas-goenka commented Jul 27, 2026

View reviewed changes

shreyas-goenka added 3 commits July 27, 2026 16:24

bundle: drop the dms package comment

309a1a4

Co-authored-by: Isaac

shreyas-goenka commented Jul 27, 2026

View reviewed changes

shreyas-goenka added 2 commits July 28, 2026 00:30

shreyas-goenka commented Jul 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

bundle: record and read deployment state via DMS - #6074

bundle: record and read deployment state via DMS#6074
shreyas-goenka wants to merge 14 commits into
databricks:mainfrom
shreyas-goenka:isaac/pr6052-fixes

shreyas-goenka commented Jul 27, 2026 •

edited

Loading

Uh oh!

shreyas-goenka Jul 27, 2026

Uh oh!

shreyas-goenka Jul 27, 2026

Uh oh!

shreyas-goenka Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

shreyas-goenka commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it does

Only for net-new deployments

Notes for reviewers

Tests

Uh oh!

shreyas-goenka Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

shreyas-goenka commented Jul 27, 2026 •

edited

Loading