Skip to content

release: Run promote finalize once, and make tag creation retry-safe - #979

Merged
DaedalusG merged 1 commit into
mainfrom
wg/promote-finalize-once
Oct 24, 2026
Merged

DaedalusG merged 1 commit into
mainfrom
wg/promote-finalize-once

Conversation

@DaedalusG

@DaedalusG DaedalusG commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

The v8.1.1335 promotion ran finalize twice, and the run the release worker watches lost the race:

Build Commit Started by Promote finalize
#3093 4d8b729 "release: v8.1.1335" (internal config commit) creating promote/release-v8.1.1335 passed at 07:14: tagged the promote branch head a018051, pushed release-v8.1.1335, promoted on the release registry
#3094 21ae663 image update push test failed (no config)
#3095 a018051 "promote-release: v8.1.1335" push failed 4×: fatal: tag 'v8.1.1335' already exists

Creating the promote branch from the internal branch builds the internal config commit, and --config-from-commit accepts its config, so that build runs the promote jobs too. Its finalize tags whatever the promote branch head is when it gets there. This time that was the right commit, only because the tests ahead of it took about 7 minutes. Build #3095 is the one the release worker monitors (sourcegraph/sourcegraph#17343), so the promote workflow failed and never closed the release PRs or opened the "Update main" PR. The release itself is fine: tag v8.1.1335 is on a018051 with public images, and the registry shows it as public. v8.0.0 had the same race (#2990 passed finalize, #2993 failed it), but nothing monitored the promote build then.

This PR changes two things:

  • Promote jobs run only for the promote config commit. (promote) Release: test and finalize now also require build.message =~ /^promote-release:/. Both the release worker and release.yaml's manual flow write that message.
  • git:tag is retry-safe. If vX already points at the promote branch head, it skips creating the tag and continues. If it points anywhere else, it fails. The release registry's promote endpoint already treats re-promoting as a no-op (200), so a retried finalize now succeeds end to end.

How this was tested

  • Job conditions. I checked the old and new conditions against the real branch and commit message of each v8.1.1335 build:

    Build Promote jobs before after
    #3093 release: v8.1.1335 run skip
    #3094 Update Helm chart images… run skip
    #3095 promote-release: v8.1.1335 run run
    Update main: build v8.1.1335 #978 promote/…-update-main skip skip

    This PR's own Buildkite build validates that the pipeline, with its new conditionals, uploads.

  • git:tag. I extracted the step from release.yaml, stubbed gh and buildkite-agent, and ran it in fresh clones against a scratch bare remote:

    • First finalize: exits 0; the tag and release-vX are created on the promote head.
    • Retried or duplicate finalize (tag already on the promote head): exits 0 with "already points at … not recreating it"; refs unchanged.
    • Tag on a different commit: exits 1 with "already exists at , not at promote/release-vX ()".
    • Control: main's step on the retry case exits 128 with fatal: tag 'v9.9.9' already exists, the v8.1.1335 failure.

Rollout

Buildkite reads pipeline.yaml and release.yaml from the commit it builds, so patch releases need this on their X.Y.x branch. It's labelled backport 8.1.x, which applies cleanly. 8.0.x predates the -update-main exclusion, so it would need a manual backport if it gets more patches.

The same change is open for the other deploy repos: sourcegraph/deploy-sourcegraph-k8s#497 (k8s failed v8.1.1335 the same way) and sourcegraph/deploy-sourcegraph-docker#1449 (Docker avoided it only because the extra build was cancelled).

The v8.1.1335 cleanup is done by hand: #976 and #977 are closed, and #978 is the "Update main" PR.

Creating the promote branch builds its first commit, the internal config
commit "release: vX". `--config-from-commit` accepts that config, so the
build runs the promote test and finalize jobs too. For v8.1.1335, that
build's finalize tagged the promote branch head and promoted the release.
Five minutes later, the build the release worker monitors, for the
"promote-release: vX" commit, failed finalize with "tag already exists",
and the promote workflow failed. v8.0.0 had the same race; nothing
monitored the promote build until the worker started doing so.

Run the promote jobs only when the commit message starts with
"promote-release:", the message both the release worker and release.yaml's
manual flow use for the promote config commit. Make git:tag skip creating
the tag when it already points at the promote branch head, and fail only
when it points elsewhere, so a retried finalize succeeds. The release
registry already treats re-promoting a release as a no-op.

Amp-Thread-ID: https://ampcode.com/threads/T-01a13c8e-c52e-7584-ab27-1a7fb05bc3b7
@DaedalusG DaedalusG added the backport 8.1.x Backport to 8.1.x release branch label Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

@DaedalusG
DaedalusG merged commit ebaf2a2 into main Oct 24, 2026
8 checks passed
@DaedalusG
DaedalusG deleted the wg/promote-finalize-once branch October 24, 2026 04:51
DaedalusG added a commit to sourcegraph/deploy-sourcegraph-k8s that referenced this pull request Oct 24, 2026
…497)

The v8.1.1335 promotion ran finalize twice in this repo, and the build
the release worker monitors lost:

| Build | Commit | Started by | Promote finalize |
|---|---|---|---|
|
[#1197](https://buildkite.com/sourcegraph/deploy-sourcegraph-k8s/builds/1197)
| `e84e57b` "release: v8.1.1335" (internal config commit) | creating
`promote/release-v8.1.1335` | **passed** at 07:09: tagged the promote
branch head `6a0aebb`, pushed `release-v8.1.1335`, promoted on the
release registry |
|
[#1201](https://buildkite.com/sourcegraph/deploy-sourcegraph-k8s/builds/1201)
| `6a0aebb` "promote-release: v8.1.1335" | push | **failed** 4×: `fatal:
tag 'v8.1.1335' already exists` |

Creating the promote branch from the internal branch builds the internal
config commit, and `--config-from-commit` accepts its config, so that
build runs the promote jobs too. Its finalize tags whatever the promote
branch head is when it gets there. Build #1201 is the one the release
worker monitors, so the promote workflow failed and never closed the
release PRs or opened the "Update main" PR. Those were done by hand:
#494 and #495 are closed, and #496 is the "Update main" PR. Helm failed
the same way (sourcegraph/deploy-sourcegraph-helm#979). Docker avoided
it only because the extra build was cancelled by a newer push.

This is the same change as sourcegraph/deploy-sourcegraph-helm#979 and
sourcegraph/deploy-sourcegraph-docker#1449:

- **Promote jobs run only for the promote config commit.** `(promote)
Release: test` and `finalize` now also require `build.message =~
/^promote-release:/`. Both the release worker and this repo's
`release.yaml` manual flow write that message.
- **`git:tag` is retry-safe.** If `vX` already points at the promote
branch head, it skips creating the tag and continues. If it points
anywhere else, it fails. The release registry's promote endpoint already
treats re-promoting as a no-op (200), so a retried finalize now succeeds
end to end.

## How this was tested

- **Job conditions.** I checked the old and new conditions against this
repo's real v8.1.1335 promote-branch commits. Before, every commit ran
the promote jobs. After, only `6a0aebb` "promote-release: v8.1.1335"
runs them; `e84e57b` "release: v8.1.1335" (build #1197) and the three
image-update commits skip them. This PR's Buildkite build validates that
the pipeline, with its new conditionals, uploads.
- **`git:tag`.** I extracted the step from this repo's `release.yaml`,
stubbed `gh` and `buildkite-agent`, and ran it in fresh clones against a
scratch bare remote:
- **First finalize:** exits 0; the tag and `release-vX` are on the
promote head.
- **Retry or duplicate:** exits 0 with "already points at … not
recreating it".
- **Tag on another commit:** exits 1 with a message naming both commits.
- **Control:** `main`'s step on the retry case exits 128 with `fatal:
tag 'v9.9.9' already exists`.

## Rollout

Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it
builds, so patch releases need this on their `X.Y.x` branch. It's
labelled `backport 8.1.x`; both files are identical on `8.1.x`, so the
backport applies cleanly.
@sourcegraph-release-bot

Copy link
Copy Markdown
Collaborator

The backport to 8.1.x failed at https://github.com/sourcegraph/deploy-sourcegraph-helm/actions/runs/40064001079:

Reviews may only be requested from collaborators. One or more of the teams you specified is not a collaborator of the sourcegraph/deploy-sourcegraph-helm repository.

To backport this PR manually, you can either:

Via the sg tool

Use the sg backport command to backport your commit to the release branch.

sg backport -r 8.1.x -p 979
Via your terminal

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-8.1.x 8.1.x
# Navigate to the new working tree
cd .worktrees/backport-8.1.x
# Create a new branch
git switch --create backport-979-to-8.1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ebaf2a25ef3988adce900e488dbb6690c7049387
# Push it to GitHub
git push --set-upstream origin backport-979-to-8.1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-8.1.x

If you encouter conflict, first resolve the conflict and stage all files, then run the commands below:

git cherry-pick --continue
# Push it to GitHub
git push --set-upstream origin backport-979-to-8.1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-8.1.x
  • Follow above instructions to backport the commit.
  • Create a pull request where the base branch is 8.1.x and the compare/head branch is backport-979-to-8.1.x., click here to create the pull request.

Once the pull request has been created, please ensure the following:

  • Make sure to tag @sourcegraph/release in the pull request description.

  • kindly remove the release-blocker from this pull request.

DaedalusG added a commit to sourcegraph/deploy-sourcegraph-docker that referenced this pull request Oct 24, 2026
…1449)

In v8.1.1335, the k8s and Helm repos both failed their promotion the
same way: creating the promote branch builds its first commit, the
internal config commit "release: vX". `--config-from-commit` accepts
that config, so the build runs the promote test and finalize jobs too.
Its finalize tags the promote branch head, and the build the release
worker monitors, for the "promote-release: vX" commit, then fails
finalize with `fatal: tag 'vX' already exists`. The promote workflow
fails and skips closing the release PRs and opening "Update main". See
sourcegraph/deploy-sourcegraph-k8s builds #1197/#1201 and
sourcegraph/deploy-sourcegraph-helm builds #3093/#3095.

This repo has the same pipeline conditions and `git:tag` step. It got
through v8.1.1335 only because the build of the internal config commit
([#16700](https://buildkite.com/sourcegraph/deploy-sourcegraph-docker/builds/16700))
was cancelled 10 seconds after it started, when newer commits arrived.
That depends on a pipeline setting, not on anything in this repo.

This is the same change as sourcegraph/deploy-sourcegraph-helm#979 and
sourcegraph/deploy-sourcegraph-k8s#497:

- **Promote jobs run only for the promote config commit.** `(promote)
Release: test` and `finalize` now also require `build.message =~
/^promote-release:/`. Both the release worker and this repo's
`release.yaml` manual flow write that message.
- **`git:tag` is retry-safe.** If `vX` already points at the promote
branch head, it skips creating the tag and continues. If it points
anywhere else, it fails. The release registry's promote endpoint already
treats re-promoting as a no-op (200), so a retried finalize now succeeds
end to end.

## How this was tested

- **Job conditions.** I checked the old and new conditions against this
repo's real v8.1.1335 promote-branch commits. Before, every commit ran
the promote jobs. After, only `63f7963` "promote-release: v8.1.1335"
runs them; `8f0974d` "release: v8.1.1335" and `6845a34` (image update)
skip them. This PR's Buildkite build validates that the pipeline, with
its new conditionals, uploads.
- **`git:tag`.** I extracted the step from this repo's `release.yaml`,
stubbed `gh` and `buildkite-agent`, and ran it in fresh clones against a
scratch bare remote:
- **First finalize:** exits 0; the tag and `release-vX` are on the
promote head.
- **Retry or duplicate:** exits 0 with "already points at … not
recreating it".
- **Tag on another commit:** exits 1 with a message naming both commits.
- **Control:** `main`'s step on the retry case exits 128 with `fatal:
tag 'v9.9.9' already exists`.

## Rollout

Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it
builds, so patch releases need this on their `X.Y.x` branch. It's
labelled `backport 8.1.x`; both files are identical on `8.1.x`, so the
backport applies cleanly.
DaedalusG added a commit that referenced this pull request Oct 24, 2026
…ation retry-safe (#981)

Backport ebaf2a2 from #979

The v8.1.1335 promotion ran finalize twice, and the run the release
worker watches lost the race:

| Build | Commit | Started by | Promote finalize |
|---|---|---|---|
|
[#3093](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3093)
| `4d8b729` "release: v8.1.1335" (internal config commit) | creating
`promote/release-v8.1.1335` | **passed** at 07:14: tagged the promote
branch head `a018051`, pushed `release-v8.1.1335`, promoted on the
release registry |
|
[#3094](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3094)
| `21ae663` image update | push | test failed (no config) |
|
[#3095](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3095)
| `a018051` "promote-release: v8.1.1335" | push | **failed** 4×: `fatal:
tag 'v8.1.1335' already exists` |

Creating the promote branch from the internal branch builds the internal
config commit, and `--config-from-commit` accepts its config, so that
build runs the promote jobs too. Its finalize tags whatever the promote
branch head is when it gets there. This time that was the right commit,
only because the tests ahead of it took about 7 minutes. Build #3095 is
the one the release worker monitors (sourcegraph/sourcegraph#17343), so
the promote workflow failed and never closed the release PRs or opened
the "Update main" PR. The release itself is fine: tag `v8.1.1335` is on
`a018051` with public images, and the registry shows it as public.
v8.0.0 had the same race (#2990 passed finalize, #2993 failed it), but
nothing monitored the promote build then.

This PR changes two things:

- **Promote jobs run only for the promote config commit.** `(promote)
Release: test` and `finalize` now also require `build.message =~
/^promote-release:/`. Both the release worker and `release.yaml`'s
manual flow write that message.
- **`git:tag` is retry-safe.** If `vX` already points at the promote
branch head, it skips creating the tag and continues. If it points
anywhere else, it fails. The release registry's promote endpoint already
treats re-promoting as a no-op (200), so a retried finalize now succeeds
end to end.

## How this was tested

- **Job conditions.** I checked the old and new conditions against the
real branch and commit message of each v8.1.1335 build:

  | Build | Promote jobs before | after |
  |---|---|---|
  | #3093 `release: v8.1.1335` | run | **skip** |
  | #3094 `Update Helm chart images…` | run | **skip** |
  | #3095 `promote-release: v8.1.1335` | run | run |
  | #978 `promote/…-update-main` | skip | skip |

This PR's own Buildkite build validates that the pipeline, with its new
conditionals, uploads.
- **`git:tag`.** I extracted the step from `release.yaml`, stubbed `gh`
and `buildkite-agent`, and ran it in fresh clones against a scratch bare
remote:
- **First finalize:** exits 0; the tag and `release-vX` are created on
the promote head.
- **Retried or duplicate finalize** (tag already on the promote head):
exits 0 with "already points at … not recreating it"; refs unchanged.
- **Tag on a different commit:** exits 1 with "already exists at <sha>,
not at promote/release-vX (<sha>)".
- **Control:** `main`'s step on the retry case exits 128 with `fatal:
tag 'v9.9.9' already exists`, the v8.1.1335 failure.

## Rollout

Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it
builds, so patch releases need this on their `X.Y.x` branch. It's
labelled `backport 8.1.x`, which applies cleanly. `8.0.x` predates the
`-update-main` exclusion, so it would need a manual backport if it gets
more patches.

The same change is open for the other deploy repos:
sourcegraph/deploy-sourcegraph-k8s#497 (k8s failed v8.1.1335 the same
way) and sourcegraph/deploy-sourcegraph-docker#1449 (Docker avoided it
only because the extra build was cancelled).

The v8.1.1335 cleanup is done by hand: #976 and #977 are closed, and
#978 is the "Update main" PR.

Co-authored-by: Warren Gifford <warrenbruceg@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants