Repository navigation
Conversation
DaedalusG
enabled auto-merge (squash)
October 22, 2026 08:15
DaedalusG
added a commit
that referenced
this pull request
Oct 24, 2026
…979) The v8.1.1335 promotion ran finalize twice, and the run the release worker watches lost the race: | Build | Commit | Started by | Promote finalize | |---|---|---|---| | [#3093](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3093) | `4d8b729` "release: v8.1.1335" (internal config commit) | creating `promote/release-v8.1.1335` | **passed** at 07:14: tagged the promote branch head `a018051`, pushed `release-v8.1.1335`, promoted on the release registry | | [#3094](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3094) | `21ae663` image update | push | test failed (no config) | | [#3095](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3095) | `a018051` "promote-release: v8.1.1335" | push | **failed** 4×: `fatal: tag 'v8.1.1335' already exists` | Creating the promote branch from the internal branch builds the internal config commit, and `--config-from-commit` accepts its config, so that build runs the promote jobs too. Its finalize tags whatever the promote branch head is when it gets there. This time that was the right commit, only because the tests ahead of it took about 7 minutes. Build #3095 is the one the release worker monitors (sourcegraph/sourcegraph#17343), so the promote workflow failed and never closed the release PRs or opened the "Update main" PR. The release itself is fine: tag `v8.1.1335` is on `a018051` with public images, and the registry shows it as public. v8.0.0 had the same race (#2990 passed finalize, #2993 failed it), but nothing monitored the promote build then. This PR changes two things: - **Promote jobs run only for the promote config commit.** `(promote) Release: test` and `finalize` now also require `build.message =~ /^promote-release:/`. Both the release worker and `release.yaml`'s manual flow write that message. - **`git:tag` is retry-safe.** If `vX` already points at the promote branch head, it skips creating the tag and continues. If it points anywhere else, it fails. The release registry's promote endpoint already treats re-promoting as a no-op (200), so a retried finalize now succeeds end to end. ## How this was tested - **Job conditions.** I checked the old and new conditions against the real branch and commit message of each v8.1.1335 build: | Build | Promote jobs before | after | |---|---|---| | #3093 `release: v8.1.1335` | run | **skip** | | #3094 `Update Helm chart images…` | run | **skip** | | #3095 `promote-release: v8.1.1335` | run | run | | #978 `promote/…-update-main` | skip | skip | This PR's own Buildkite build validates that the pipeline, with its new conditionals, uploads. - **`git:tag`.** I extracted the step from `release.yaml`, stubbed `gh` and `buildkite-agent`, and ran it in fresh clones against a scratch bare remote: - **First finalize:** exits 0; the tag and `release-vX` are created on the promote head. - **Retried or duplicate finalize** (tag already on the promote head): exits 0 with "already points at … not recreating it"; refs unchanged. - **Tag on a different commit:** exits 1 with "already exists at <sha>, not at promote/release-vX (<sha>)". - **Control:** `main`'s step on the retry case exits 128 with `fatal: tag 'v9.9.9' already exists`, the v8.1.1335 failure. ## Rollout Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it builds, so patch releases need this on their `X.Y.x` branch. It's labelled `backport 8.1.x`, which applies cleanly. `8.0.x` predates the `-update-main` exclusion, so it would need a manual backport if it gets more patches. The same change is open for the other deploy repos: sourcegraph/deploy-sourcegraph-k8s#497 (k8s failed v8.1.1335 the same way) and sourcegraph/deploy-sourcegraph-docker#1449 (Docker avoided it only because the extra build was cancelled). The v8.1.1335 cleanup is done by hand: #976 and #977 are closed, and #978 is the "Update main" PR.
DaedalusG
added a commit
that referenced
this pull request
Oct 24, 2026
…ation retry-safe (#981) Backport ebaf2a2 from #979 The v8.1.1335 promotion ran finalize twice, and the run the release worker watches lost the race: | Build | Commit | Started by | Promote finalize | |---|---|---|---| | [#3093](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3093) | `4d8b729` "release: v8.1.1335" (internal config commit) | creating `promote/release-v8.1.1335` | **passed** at 07:14: tagged the promote branch head `a018051`, pushed `release-v8.1.1335`, promoted on the release registry | | [#3094](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3094) | `21ae663` image update | push | test failed (no config) | | [#3095](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3095) | `a018051` "promote-release: v8.1.1335" | push | **failed** 4×: `fatal: tag 'v8.1.1335' already exists` | Creating the promote branch from the internal branch builds the internal config commit, and `--config-from-commit` accepts its config, so that build runs the promote jobs too. Its finalize tags whatever the promote branch head is when it gets there. This time that was the right commit, only because the tests ahead of it took about 7 minutes. Build #3095 is the one the release worker monitors (sourcegraph/sourcegraph#17343), so the promote workflow failed and never closed the release PRs or opened the "Update main" PR. The release itself is fine: tag `v8.1.1335` is on `a018051` with public images, and the registry shows it as public. v8.0.0 had the same race (#2990 passed finalize, #2993 failed it), but nothing monitored the promote build then. This PR changes two things: - **Promote jobs run only for the promote config commit.** `(promote) Release: test` and `finalize` now also require `build.message =~ /^promote-release:/`. Both the release worker and `release.yaml`'s manual flow write that message. - **`git:tag` is retry-safe.** If `vX` already points at the promote branch head, it skips creating the tag and continues. If it points anywhere else, it fails. The release registry's promote endpoint already treats re-promoting as a no-op (200), so a retried finalize now succeeds end to end. ## How this was tested - **Job conditions.** I checked the old and new conditions against the real branch and commit message of each v8.1.1335 build: | Build | Promote jobs before | after | |---|---|---| | #3093 `release: v8.1.1335` | run | **skip** | | #3094 `Update Helm chart images…` | run | **skip** | | #3095 `promote-release: v8.1.1335` | run | run | | #978 `promote/…-update-main` | skip | skip | This PR's own Buildkite build validates that the pipeline, with its new conditionals, uploads. - **`git:tag`.** I extracted the step from `release.yaml`, stubbed `gh` and `buildkite-agent`, and ran it in fresh clones against a scratch bare remote: - **First finalize:** exits 0; the tag and `release-vX` are created on the promote head. - **Retried or duplicate finalize** (tag already on the promote head): exits 0 with "already points at … not recreating it"; refs unchanged. - **Tag on a different commit:** exits 1 with "already exists at <sha>, not at promote/release-vX (<sha>)". - **Control:** `main`'s step on the retry case exits 128 with `fatal: tag 'v9.9.9' already exists`, the v8.1.1335 failure. ## Rollout Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it builds, so patch releases need this on their `X.Y.x` branch. It's labelled `backport 8.1.x`, which applies cleanly. `8.0.x` predates the `-update-main` exclusion, so it would need a manual backport if it gets more patches. The same change is open for the other deploy repos: sourcegraph/deploy-sourcegraph-k8s#497 (k8s failed v8.1.1335 the same way) and sourcegraph/deploy-sourcegraph-docker#1449 (Docker avoided it only because the extra build was cancelled). The v8.1.1335 cleanup is done by hand: #976 and #977 are closed, and #978 is the "Update main" PR. Co-authored-by: Warren Gifford <warrenbruceg@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test plan: automated release PR, CI will perform additional checks
Opened by hand. The release workflow normally opens this PR as its last step, but its promote child failed while monitoring build #3095 (a duplicate finalize; fixed by #979), so it never got here.
The commits match what the workflow's
openUpdateMainPRproduces: every pinned image in the four charts is rewritten to the public8.1.1335digest, and the fiveChart.yamlfiles are bumped to8.1.1335. The digests come from thev8.1.1335tag, and each chart's image map now equals the tag's exactly. No README changes.Supersedes #961 (Update main: build v8.1.0).