Repository navigation
release: Run promote finalize once, and make tag creation retry-safe - #497
Merged
Merged
Conversation
Creating the promote branch builds its first commit, the internal config commit "release: vX". `--config-from-commit` accepts that config, so the build runs the promote test and finalize jobs too, and its finalize tags the promote branch head. The build the release worker monitors, for the "promote-release: vX" commit, then fails finalize with "tag already exists", and the promote workflow fails. v8.1.1335 hit this in the k8s and Helm repos; docker only avoided it because the extra build was cancelled by a newer push. Run the promote jobs only when the commit message starts with "promote-release:", the message both the release worker and release.yaml's manual flow use for the promote config commit. Make git:tag skip creating the tag when it already points at the promote branch head, and fail only when it points elsewhere, so a retried finalize succeeds. The release registry already treats re-promoting a release as a no-op. Same change as sourcegraph/deploy-sourcegraph-helm#979. Amp-Thread-ID: https://ampcode.com/threads/T-01a13c8e-c52e-7584-ab27-1a7fb05bc3b7
devdinu
approved these changes
Oct 23, 2026
DaedalusG
added a commit
to sourcegraph/deploy-sourcegraph-helm
that referenced
this pull request
Oct 24, 2026
…979) The v8.1.1335 promotion ran finalize twice, and the run the release worker watches lost the race: | Build | Commit | Started by | Promote finalize | |---|---|---|---| | [#3093](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3093) | `4d8b729` "release: v8.1.1335" (internal config commit) | creating `promote/release-v8.1.1335` | **passed** at 07:14: tagged the promote branch head `a018051`, pushed `release-v8.1.1335`, promoted on the release registry | | [#3094](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3094) | `21ae663` image update | push | test failed (no config) | | [#3095](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3095) | `a018051` "promote-release: v8.1.1335" | push | **failed** 4×: `fatal: tag 'v8.1.1335' already exists` | Creating the promote branch from the internal branch builds the internal config commit, and `--config-from-commit` accepts its config, so that build runs the promote jobs too. Its finalize tags whatever the promote branch head is when it gets there. This time that was the right commit, only because the tests ahead of it took about 7 minutes. Build #3095 is the one the release worker monitors (sourcegraph/sourcegraph#17343), so the promote workflow failed and never closed the release PRs or opened the "Update main" PR. The release itself is fine: tag `v8.1.1335` is on `a018051` with public images, and the registry shows it as public. v8.0.0 had the same race (#2990 passed finalize, #2993 failed it), but nothing monitored the promote build then. This PR changes two things: - **Promote jobs run only for the promote config commit.** `(promote) Release: test` and `finalize` now also require `build.message =~ /^promote-release:/`. Both the release worker and `release.yaml`'s manual flow write that message. - **`git:tag` is retry-safe.** If `vX` already points at the promote branch head, it skips creating the tag and continues. If it points anywhere else, it fails. The release registry's promote endpoint already treats re-promoting as a no-op (200), so a retried finalize now succeeds end to end. ## How this was tested - **Job conditions.** I checked the old and new conditions against the real branch and commit message of each v8.1.1335 build: | Build | Promote jobs before | after | |---|---|---| | #3093 `release: v8.1.1335` | run | **skip** | | #3094 `Update Helm chart images…` | run | **skip** | | #3095 `promote-release: v8.1.1335` | run | run | | #978 `promote/…-update-main` | skip | skip | This PR's own Buildkite build validates that the pipeline, with its new conditionals, uploads. - **`git:tag`.** I extracted the step from `release.yaml`, stubbed `gh` and `buildkite-agent`, and ran it in fresh clones against a scratch bare remote: - **First finalize:** exits 0; the tag and `release-vX` are created on the promote head. - **Retried or duplicate finalize** (tag already on the promote head): exits 0 with "already points at … not recreating it"; refs unchanged. - **Tag on a different commit:** exits 1 with "already exists at <sha>, not at promote/release-vX (<sha>)". - **Control:** `main`'s step on the retry case exits 128 with `fatal: tag 'v9.9.9' already exists`, the v8.1.1335 failure. ## Rollout Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it builds, so patch releases need this on their `X.Y.x` branch. It's labelled `backport 8.1.x`, which applies cleanly. `8.0.x` predates the `-update-main` exclusion, so it would need a manual backport if it gets more patches. The same change is open for the other deploy repos: sourcegraph/deploy-sourcegraph-k8s#497 (k8s failed v8.1.1335 the same way) and sourcegraph/deploy-sourcegraph-docker#1449 (Docker avoided it only because the extra build was cancelled). The v8.1.1335 cleanup is done by hand: #976 and #977 are closed, and #978 is the "Update main" PR.
DaedalusG
added a commit
to sourcegraph/deploy-sourcegraph-docker
that referenced
this pull request
Oct 24, 2026
…1449) In v8.1.1335, the k8s and Helm repos both failed their promotion the same way: creating the promote branch builds its first commit, the internal config commit "release: vX". `--config-from-commit` accepts that config, so the build runs the promote test and finalize jobs too. Its finalize tags the promote branch head, and the build the release worker monitors, for the "promote-release: vX" commit, then fails finalize with `fatal: tag 'vX' already exists`. The promote workflow fails and skips closing the release PRs and opening "Update main". See sourcegraph/deploy-sourcegraph-k8s builds #1197/#1201 and sourcegraph/deploy-sourcegraph-helm builds #3093/#3095. This repo has the same pipeline conditions and `git:tag` step. It got through v8.1.1335 only because the build of the internal config commit ([#16700](https://buildkite.com/sourcegraph/deploy-sourcegraph-docker/builds/16700)) was cancelled 10 seconds after it started, when newer commits arrived. That depends on a pipeline setting, not on anything in this repo. This is the same change as sourcegraph/deploy-sourcegraph-helm#979 and sourcegraph/deploy-sourcegraph-k8s#497: - **Promote jobs run only for the promote config commit.** `(promote) Release: test` and `finalize` now also require `build.message =~ /^promote-release:/`. Both the release worker and this repo's `release.yaml` manual flow write that message. - **`git:tag` is retry-safe.** If `vX` already points at the promote branch head, it skips creating the tag and continues. If it points anywhere else, it fails. The release registry's promote endpoint already treats re-promoting as a no-op (200), so a retried finalize now succeeds end to end. ## How this was tested - **Job conditions.** I checked the old and new conditions against this repo's real v8.1.1335 promote-branch commits. Before, every commit ran the promote jobs. After, only `63f7963` "promote-release: v8.1.1335" runs them; `8f0974d` "release: v8.1.1335" and `6845a34` (image update) skip them. This PR's Buildkite build validates that the pipeline, with its new conditionals, uploads. - **`git:tag`.** I extracted the step from this repo's `release.yaml`, stubbed `gh` and `buildkite-agent`, and ran it in fresh clones against a scratch bare remote: - **First finalize:** exits 0; the tag and `release-vX` are on the promote head. - **Retry or duplicate:** exits 0 with "already points at … not recreating it". - **Tag on another commit:** exits 1 with a message naming both commits. - **Control:** `main`'s step on the retry case exits 128 with `fatal: tag 'v9.9.9' already exists`. ## Rollout Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it builds, so patch releases need this on their `X.Y.x` branch. It's labelled `backport 8.1.x`; both files are identical on `8.1.x`, so the backport applies cleanly.
DaedalusG
added a commit
to sourcegraph/deploy-sourcegraph-helm
that referenced
this pull request
Oct 24, 2026
…ation retry-safe (#981) Backport ebaf2a2 from #979 The v8.1.1335 promotion ran finalize twice, and the run the release worker watches lost the race: | Build | Commit | Started by | Promote finalize | |---|---|---|---| | [#3093](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3093) | `4d8b729` "release: v8.1.1335" (internal config commit) | creating `promote/release-v8.1.1335` | **passed** at 07:14: tagged the promote branch head `a018051`, pushed `release-v8.1.1335`, promoted on the release registry | | [#3094](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3094) | `21ae663` image update | push | test failed (no config) | | [#3095](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3095) | `a018051` "promote-release: v8.1.1335" | push | **failed** 4×: `fatal: tag 'v8.1.1335' already exists` | Creating the promote branch from the internal branch builds the internal config commit, and `--config-from-commit` accepts its config, so that build runs the promote jobs too. Its finalize tags whatever the promote branch head is when it gets there. This time that was the right commit, only because the tests ahead of it took about 7 minutes. Build #3095 is the one the release worker monitors (sourcegraph/sourcegraph#17343), so the promote workflow failed and never closed the release PRs or opened the "Update main" PR. The release itself is fine: tag `v8.1.1335` is on `a018051` with public images, and the registry shows it as public. v8.0.0 had the same race (#2990 passed finalize, #2993 failed it), but nothing monitored the promote build then. This PR changes two things: - **Promote jobs run only for the promote config commit.** `(promote) Release: test` and `finalize` now also require `build.message =~ /^promote-release:/`. Both the release worker and `release.yaml`'s manual flow write that message. - **`git:tag` is retry-safe.** If `vX` already points at the promote branch head, it skips creating the tag and continues. If it points anywhere else, it fails. The release registry's promote endpoint already treats re-promoting as a no-op (200), so a retried finalize now succeeds end to end. ## How this was tested - **Job conditions.** I checked the old and new conditions against the real branch and commit message of each v8.1.1335 build: | Build | Promote jobs before | after | |---|---|---| | #3093 `release: v8.1.1335` | run | **skip** | | #3094 `Update Helm chart images…` | run | **skip** | | #3095 `promote-release: v8.1.1335` | run | run | | #978 `promote/…-update-main` | skip | skip | This PR's own Buildkite build validates that the pipeline, with its new conditionals, uploads. - **`git:tag`.** I extracted the step from `release.yaml`, stubbed `gh` and `buildkite-agent`, and ran it in fresh clones against a scratch bare remote: - **First finalize:** exits 0; the tag and `release-vX` are created on the promote head. - **Retried or duplicate finalize** (tag already on the promote head): exits 0 with "already points at … not recreating it"; refs unchanged. - **Tag on a different commit:** exits 1 with "already exists at <sha>, not at promote/release-vX (<sha>)". - **Control:** `main`'s step on the retry case exits 128 with `fatal: tag 'v9.9.9' already exists`, the v8.1.1335 failure. ## Rollout Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it builds, so patch releases need this on their `X.Y.x` branch. It's labelled `backport 8.1.x`, which applies cleanly. `8.0.x` predates the `-update-main` exclusion, so it would need a manual backport if it gets more patches. The same change is open for the other deploy repos: sourcegraph/deploy-sourcegraph-k8s#497 (k8s failed v8.1.1335 the same way) and sourcegraph/deploy-sourcegraph-docker#1449 (Docker avoided it only because the extra build was cancelled). The v8.1.1335 cleanup is done by hand: #976 and #977 are closed, and #978 is the "Update main" PR. Co-authored-by: Warren Gifford <warrenbruceg@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v8.1.1335 promotion ran finalize twice in this repo, and the build the release worker monitors lost:
e84e57b"release: v8.1.1335" (internal config commit)promote/release-v8.1.13356a0aebb, pushedrelease-v8.1.1335, promoted on the release registry6a0aebb"promote-release: v8.1.1335"fatal: tag 'v8.1.1335' already existsCreating the promote branch from the internal branch builds the internal config commit, and
--config-from-commitaccepts its config, so that build runs the promote jobs too. Its finalize tags whatever the promote branch head is when it gets there. Build #1201 is the one the release worker monitors, so the promote workflow failed and never closed the release PRs or opened the "Update main" PR. Those were done by hand: #494 and #495 are closed, and #496 is the "Update main" PR. Helm failed the same way (sourcegraph/deploy-sourcegraph-helm#979). Docker avoided it only because the extra build was cancelled by a newer push.This is the same change as sourcegraph/deploy-sourcegraph-helm#979 and sourcegraph/deploy-sourcegraph-docker#1449:
(promote) Release: testandfinalizenow also requirebuild.message =~ /^promote-release:/. Both the release worker and this repo'srelease.yamlmanual flow write that message.git:tagis retry-safe. IfvXalready points at the promote branch head, it skips creating the tag and continues. If it points anywhere else, it fails. The release registry's promote endpoint already treats re-promoting as a no-op (200), so a retried finalize now succeeds end to end.How this was tested
6a0aebb"promote-release: v8.1.1335" runs them;e84e57b"release: v8.1.1335" (build #1197) and the three image-update commits skip them. This PR's Buildkite build validates that the pipeline, with its new conditionals, uploads.git:tag. I extracted the step from this repo'srelease.yaml, stubbedghandbuildkite-agent, and ran it in fresh clones against a scratch bare remote:release-vXare on the promote head.main's step on the retry case exits 128 withfatal: tag 'v9.9.9' already exists.Rollout
Buildkite reads
pipeline.yamlandrelease.yamlfrom the commit it builds, so patch releases need this on theirX.Y.xbranch. It's labelledbackport 8.1.x; both files are identical on8.1.x, so the backport applies cleanly.