Skip to content

Fix pipeline allow_duplicate_names never converges - #6076

Open
radakam wants to merge 1 commit into
mainfrom
fix-pipeline-allow-duplicate-names-drift
Open

Fix pipeline allow_duplicate_names never converges#6076
radakam wants to merge 1 commit into
mainfrom
fix-pipeline-allow-duplicate-names-drift

Conversation

@radakam

@radakam radakam commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Changes

Declare pipeline allow_duplicate_names as input_only in ignore_remote_changes, next to the existing run_as entry. Adds a no_drift invariant config covering it.

Why

The field is only carried by CreatePipeline/EditPipeline and is never returned by GET, so remote always reads back false. With allow_duplicate_names: true, every plan after a deploy reported a perpetual update (old: true, new: true — config unchanged, phantom remote drift). It's in RemoteType via the embedded CreatePipeline, so the automatic missing_in_remote suppression doesn't apply.

dry_run is request-only in the same way, but EditPipeline has no DryRun field, so it needs its own fix and is out of scope here.

Tests

New pipeline_allow_duplicate_names.yml.tmpl invariant config: fails on main, passes with the fix. Also confirmed on dogfood — perpetual 1 to change across three deploys before the fix, control pipeline without the field clean.

allow_duplicate_names is only carried by CreatePipeline/EditPipeline; the
pipelines GET API never returns it, so remote always reads back false. With
allow_duplicate_names: true in config, every plan after a successful deploy
reported the pipeline as an update that could never converge.

The field is part of RemoteType via the embedded CreatePipeline, so the
automatic missing_in_remote suppression does not apply. Declare it as
input_only in ignore_remote_changes, next to run_as.
@radakam
radakam marked this pull request as ready for review July 27, 2026 13:14
@github-actions

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/bundle/ - needs approval

7 files changed
Suggested: @denik
Also eligible: @janniklasrose, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

/bundle/ - needs approval

Files: bundle/direct/dresources/resources.yml
Suggested: @denik
Also eligible: @janniklasrose, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/pipeline-allow-duplicate-names-converges.md
Based on git history:

  • @denik -- recent work in bundle/direct/dresources/, acceptance/bundle/invariant/configs/, acceptance/bundle/invariant/

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 6e808d6

Run: 30269119461

Env 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 5 3 607 1056 91:28
🔄​ aws windows 2 5 3 607 1054 110:59
💚​ azure linux 5 3 607 1055 102:09
💚​ azure windows 5 3 609 1053 108:56
💚​ gcp linux 2 4 606 1057 106:41
💚​ gcp windows 2 4 608 1055 116:22
10 interesting tests: 5 RECOVERED, 3 SKIP, 2 flaky
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=job_run.yml.tmpl/READPLAN= ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=job_run.yml.tmpl/READPLAN=1 ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
💚​ TestAccept/bundle/invariant/no_drift 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
8:49 gcp linux TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
8:33 gcp windows TestAccept
8:11 aws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
7:38 aws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
7:35 aws windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
7:23 azure windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
7:18 aws windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
7:01 azure windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
6:57 azure windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:55 aws windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:51 azure linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
6:49 azure linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
6:46 azure windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
6:22 azure windows TestAccept
6:15 gcp linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=catalog.yml.tmpl/READPLAN=1
5:49 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
5:15 gcp linux TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
5:04 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
4:51 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN=
4:50 aws linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
4:47 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
4:13 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=1
4:11 azure linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
4:02 gcp linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:58 aws linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
3:57 gcp windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:56 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=job_pydabs_10_tasks.yml.tmpl/READPLAN=1
3:50 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=catalog.yml.tmpl/READPLAN=1
3:50 azure linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=pipeline.yml.tmpl/READPLAN=1
3:50 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=volume.yml.tmpl/READPLAN=1
3:47 aws linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:46 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:45 gcp linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
3:44 gcp linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
3:41 aws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:41 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=1
3:40 azure linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=instance_pool.yml.tmpl/READPLAN=
3:38 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=pipeline.yml.tmpl/READPLAN=1
3:37 aws linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:36 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN=1
3:36 gcp linux TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:34 gcp linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:34 gcp windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:32 azure linux TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=job_run.yml.tmpl/READPLAN=1
3:31 gcp linux TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
3:25 gcp windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
3:25 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=volume_path_job_ref.yml.tmpl/READPLAN=1
3:24 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=schema_empty_grants.yml.tmpl/READPLAN=
3:23 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
3:23 gcp linux TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=secret_scope.yml.tmpl/READPLAN=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants