Repository navigation
(internal) release: build v8.1.1335 - #976
Closed
sourcegraph-bot-2 wants to merge 5 commits into
Closed
sourcegraph-bot-2 wants to merge 5 commits into
sourcegraph-bot-2 wants to merge 5 commits into
Conversation
…#973) Backport c36d89e from #963 Every release rewrites the image digests and registry in `values.yaml` and the chart version in `Chart.yaml`, and all three show up in the generated READMEs. That forced the release worker to regenerate the READMEs with its own copy of helm-docs. When that copy drifted from `scripts/helm-docs.sh`, `internal/release-v8.1.0` failed ":book: Verify helm-docs is up-to-date" and had to be fixed by hand. The regeneration also doesn't reach customers. The publish Action packages the chart from the first push to `promote/release-*`, before the docs commit lands, so the published `sourcegraph-8.0.0.tgz` README lists the internal registry as the default. This PR makes the READMEs independent of everything a release changes, so the release worker can drop its helm-docs step entirely (sourcegraph/sourcegraph#17056): - Every pinned image `defaultTag` gets `` # @default -- `<appVersion>@sha256:<digest>`, pinned by each release ``, and `sourcegraph.image.repository` gets `# @default -- Sourcegraph's public Docker Hub registry`, in the four charts the release updates. The two executor images also get the `# -- ` description helm-docs needs before it honors `@default`. The registry annotation deliberately avoids the literal registry string, because the release replaces that string everywhere in `values.yaml`, comments included. - The migrator README uses a `<chart-version>` placeholder instead of `{{ template "chart.version" . }}`. - A new CI step fails if any chart README renders a pinned digest or `index.docker.io/sourcegraph`. A new image added without an annotation then fails on its own PR rather than on the next release branch. It is a separate step so it doesn't conflict with #954, which rewrites the existing helm-docs step. - Published charts still list the exact values. Before `helm package`, the release and insiders publish workflows run the new `scripts/ci/render-published-readmes.sh`. It deletes the release-value `@default` annotations and reruns `./scripts/helm-docs.sh`, so each published README lists the same digests and registry as the `values.yaml` packaged with it. Nothing is committed. Because the docs are built from the commit being published, the README no longer depends on a later docs commit landing first, which was the cause of the wrong v8.0.0 README. The step is `continue-on-error`: if it fails (for example, the helm-docs download from GitHub fails), the chart still publishes with the committed placeholder README. The tradeoff is that READMEs viewed in this repository show placeholders instead of exact digests. The branch's `values.yaml` still has them, and the README in the published chart (`helm show readme`) shows them in full. The migrator's `helm upgrade` examples keep the `<chart-version>` placeholder in the published chart too. ## How this was tested - **Simulated release.** CI can't exercise a release, so I reproduced one on a scratch worktree. A temporary test in the sourcegraph repo ran the release worker's own `values.yaml` rewrite (`dev/images.UpdateHelmManifestContent`) over the four charts the release touches. It first rewrote them to the internal registry with new tags and digests (the internal phase), then back to the public registry (the promote phase). I bumped `Chart.yaml` `version`/`appVersion` with the same regex as `UpdateChartVersions`, and ran `./scripts/helm-docs.sh` after each phase. | | README lines changed after internal phase | after promote phase | |---|---|---| | `main` | 68 | 60 | | this branch | 0 | 0 | `values.yaml` and `Chart.yaml` changed as expected in both runs. Zero drift on this branch also shows that the release rewrite leaves the `# @default --` comments intact. - **Annotation coverage.** After regenerating, no chart README contains a 64-character `@sha256:` digest or `index.docker.io/sourcegraph`. A second `./scripts/helm-docs.sh` run leaves the tree clean. - **New CI guard.** I ran the new step's command against both trees. It exits 0 on this branch. On `main` it exits 1 and lists the offending lines, for example `sourcegraph.image.repository` in `charts/sourcegraph-executor/dind/README.md`. - **Publish rendering, end to end.** In a scratch worktree I rebuilt a promote branch the way a release would: internal phase, `Chart.yaml` bump, promote phase. Buildkite's helm-docs check stays clean on that tree. I then ran the publish job's steps with Helm v3.4.0, the version the workflows pin: the new script, then both `helm package` loops. I extracted each `.tgz` and compared its README with its packaged `values.yaml`. The expected values were taken from `values.yaml`, not from helm-docs output. | Scenario | Digests in packaged `values.yaml` | Missing from packaged README | Placeholders left | Registry row | |---|---|---|---|---| | Release publish (4 charts) | 24 / 1 / 1 / 1 | 0 | 0 | `"index.docker.io/sourcegraph"` | | Insiders publish (4 charts) | 24 / 1 / 1 / 1 | 0 | 0 | `"index.docker.io/sourcegraph"` | | Render step forced to fail | 24 / 1 / 1 / 1 | all | all | placeholder; packaging still succeeds | For every chart, in all three scenarios, `helm template` of the packaged chart matches the unmodified chart, after normalizing the migrator Job's random name suffix. The script only touches comments and READMEs. The `sourcegraph` chart has 25 annotated images but 24 distinct digests, because `codeIntelDB` and `pgsql` share one. `shellcheck` is clean, and `actionlint` reports the same three pre-existing `actions/checkout@v3` findings as on `main`. - **CI.** On the first commit, [Buildkite build 3049](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3049) passed, including ":book: Verify helm-docs is up-to-date" and the new ":book: Verify READMEs do not render release-managed values" step. "Lint and Test charts" also passed. The changes to `values.yaml` are comments only, so rendered manifests are unchanged. The publish workflows only run on pushes to `main`, `release/**` and `promote/release-**`, so this PR's CI can't exercise them. The end-to-end run above stands in for that. The first real check of the full flow will be the next release once both PRs are deployed. Merge this, and backport it to every `X.Y.x` branch that still gets patch releases (at least `8.0.x` and `8.1.x`), before the release worker change deploys. Patch releases branch from `X.Y.x`, and a branch without these annotations would fail the helm-docs check once the worker stops regenerating READMEs. Part of [PLAT-876](https://linear.app/sourcegraph/issue/PLAT-876/find-a-permanent-fix-for-helm-docs-release-failures). Co-authored-by: Warren Gifford <warrenbruceg@gmail.com>
…orker (#974) Backport 44e0288 from #972 Promote finalize opens a draft "Update main: build vX" PR after every release, but none has ever been merged, so `main` here lags many releases behind. Its Buildkite build also always fails: `promote/release-vX-update-main` matches the `^promote\/release-.*` release steps, and `(promote) Release: test` stops with "did not find serialized config" (for example, [deploy-sourcegraph-k8s build 1187](https://buildkite.com/sourcegraph/deploy-sourcegraph-k8s/builds/1187)). The releaseworker now opens the "Update main" PR itself after the promote build passes, with squash auto-merge enabled and the Platform team requested as reviewers ([sourcegraph/sourcegraph#17347](sourcegraph/sourcegraph#17347)). This PR: - removes the "Update main" steps from `promoteToPublic.finalize` in `release.yaml`, so the releaseworker and Buildkite don't both create it; - skips the `(promote) Release: test` and `(promote) Release: finalize` steps on `*-update-main` branches in `.buildkite/pipeline.yaml`. Ship this together with sourcegraph/sourcegraph#17347. Until both are live, a release either gets no "Update main" PR or a duplicate attempt that the releaseworker logs and skips. Part of [PLAT-881](https://linear.app/sourcegraph/issue/PLAT-881/close-release-generated-prs-when-release-is-complete). Matching changes: [docker#1443](sourcegraph/deploy-sourcegraph-docker#1443), [k8s#492](sourcegraph/deploy-sourcegraph-k8s#492). ### Checklist - [ ] Follow the manual testing process: not applicable, no chart changes - [ ] Update changelog: not needed, release tooling only - [ ] Update Kubernetes update doc: not needed ### Test plan Both YAML files parse, and `promoteToPublic.finalize` now contains only the tag and release registry steps. The pipeline conditions still match `promote/release-v8.1.0` and no longer match `promote/release-v8.1.0-update-main`. The next release's promote build and "Update main" PR will exercise this end to end. Co-authored-by: Warren Gifford <warrenbruceg@gmail.com>
{"version":"v8.1.1335","inputs":"server=v8.1.1335","type":"patch"}
|
|
Contributor
|
Closing by hand: v8.1.1335 is public (tag |
DaedalusG
added a commit
that referenced
this pull request
Oct 24, 2026
…979) The v8.1.1335 promotion ran finalize twice, and the run the release worker watches lost the race: | Build | Commit | Started by | Promote finalize | |---|---|---|---| | [#3093](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3093) | `4d8b729` "release: v8.1.1335" (internal config commit) | creating `promote/release-v8.1.1335` | **passed** at 07:14: tagged the promote branch head `a018051`, pushed `release-v8.1.1335`, promoted on the release registry | | [#3094](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3094) | `21ae663` image update | push | test failed (no config) | | [#3095](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3095) | `a018051` "promote-release: v8.1.1335" | push | **failed** 4×: `fatal: tag 'v8.1.1335' already exists` | Creating the promote branch from the internal branch builds the internal config commit, and `--config-from-commit` accepts its config, so that build runs the promote jobs too. Its finalize tags whatever the promote branch head is when it gets there. This time that was the right commit, only because the tests ahead of it took about 7 minutes. Build #3095 is the one the release worker monitors (sourcegraph/sourcegraph#17343), so the promote workflow failed and never closed the release PRs or opened the "Update main" PR. The release itself is fine: tag `v8.1.1335` is on `a018051` with public images, and the registry shows it as public. v8.0.0 had the same race (#2990 passed finalize, #2993 failed it), but nothing monitored the promote build then. This PR changes two things: - **Promote jobs run only for the promote config commit.** `(promote) Release: test` and `finalize` now also require `build.message =~ /^promote-release:/`. Both the release worker and `release.yaml`'s manual flow write that message. - **`git:tag` is retry-safe.** If `vX` already points at the promote branch head, it skips creating the tag and continues. If it points anywhere else, it fails. The release registry's promote endpoint already treats re-promoting as a no-op (200), so a retried finalize now succeeds end to end. ## How this was tested - **Job conditions.** I checked the old and new conditions against the real branch and commit message of each v8.1.1335 build: | Build | Promote jobs before | after | |---|---|---| | #3093 `release: v8.1.1335` | run | **skip** | | #3094 `Update Helm chart images…` | run | **skip** | | #3095 `promote-release: v8.1.1335` | run | run | | #978 `promote/…-update-main` | skip | skip | This PR's own Buildkite build validates that the pipeline, with its new conditionals, uploads. - **`git:tag`.** I extracted the step from `release.yaml`, stubbed `gh` and `buildkite-agent`, and ran it in fresh clones against a scratch bare remote: - **First finalize:** exits 0; the tag and `release-vX` are created on the promote head. - **Retried or duplicate finalize** (tag already on the promote head): exits 0 with "already points at … not recreating it"; refs unchanged. - **Tag on a different commit:** exits 1 with "already exists at <sha>, not at promote/release-vX (<sha>)". - **Control:** `main`'s step on the retry case exits 128 with `fatal: tag 'v9.9.9' already exists`, the v8.1.1335 failure. ## Rollout Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it builds, so patch releases need this on their `X.Y.x` branch. It's labelled `backport 8.1.x`, which applies cleanly. `8.0.x` predates the `-update-main` exclusion, so it would need a manual backport if it gets more patches. The same change is open for the other deploy repos: sourcegraph/deploy-sourcegraph-k8s#497 (k8s failed v8.1.1335 the same way) and sourcegraph/deploy-sourcegraph-docker#1449 (Docker avoided it only because the extra build was cancelled). The v8.1.1335 cleanup is done by hand: #976 and #977 are closed, and #978 is the "Update main" PR.
DaedalusG
added a commit
that referenced
this pull request
Oct 24, 2026
…ation retry-safe (#981) Backport ebaf2a2 from #979 The v8.1.1335 promotion ran finalize twice, and the run the release worker watches lost the race: | Build | Commit | Started by | Promote finalize | |---|---|---|---| | [#3093](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3093) | `4d8b729` "release: v8.1.1335" (internal config commit) | creating `promote/release-v8.1.1335` | **passed** at 07:14: tagged the promote branch head `a018051`, pushed `release-v8.1.1335`, promoted on the release registry | | [#3094](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3094) | `21ae663` image update | push | test failed (no config) | | [#3095](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3095) | `a018051` "promote-release: v8.1.1335" | push | **failed** 4×: `fatal: tag 'v8.1.1335' already exists` | Creating the promote branch from the internal branch builds the internal config commit, and `--config-from-commit` accepts its config, so that build runs the promote jobs too. Its finalize tags whatever the promote branch head is when it gets there. This time that was the right commit, only because the tests ahead of it took about 7 minutes. Build #3095 is the one the release worker monitors (sourcegraph/sourcegraph#17343), so the promote workflow failed and never closed the release PRs or opened the "Update main" PR. The release itself is fine: tag `v8.1.1335` is on `a018051` with public images, and the registry shows it as public. v8.0.0 had the same race (#2990 passed finalize, #2993 failed it), but nothing monitored the promote build then. This PR changes two things: - **Promote jobs run only for the promote config commit.** `(promote) Release: test` and `finalize` now also require `build.message =~ /^promote-release:/`. Both the release worker and `release.yaml`'s manual flow write that message. - **`git:tag` is retry-safe.** If `vX` already points at the promote branch head, it skips creating the tag and continues. If it points anywhere else, it fails. The release registry's promote endpoint already treats re-promoting as a no-op (200), so a retried finalize now succeeds end to end. ## How this was tested - **Job conditions.** I checked the old and new conditions against the real branch and commit message of each v8.1.1335 build: | Build | Promote jobs before | after | |---|---|---| | #3093 `release: v8.1.1335` | run | **skip** | | #3094 `Update Helm chart images…` | run | **skip** | | #3095 `promote-release: v8.1.1335` | run | run | | #978 `promote/…-update-main` | skip | skip | This PR's own Buildkite build validates that the pipeline, with its new conditionals, uploads. - **`git:tag`.** I extracted the step from `release.yaml`, stubbed `gh` and `buildkite-agent`, and ran it in fresh clones against a scratch bare remote: - **First finalize:** exits 0; the tag and `release-vX` are created on the promote head. - **Retried or duplicate finalize** (tag already on the promote head): exits 0 with "already points at … not recreating it"; refs unchanged. - **Tag on a different commit:** exits 1 with "already exists at <sha>, not at promote/release-vX (<sha>)". - **Control:** `main`'s step on the retry case exits 128 with `fatal: tag 'v9.9.9' already exists`, the v8.1.1335 failure. ## Rollout Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it builds, so patch releases need this on their `X.Y.x` branch. It's labelled `backport 8.1.x`, which applies cleanly. `8.0.x` predates the `-update-main` exclusion, so it would need a manual backport if it gets more patches. The same change is open for the other deploy repos: sourcegraph/deploy-sourcegraph-k8s#497 (k8s failed v8.1.1335 the same way) and sourcegraph/deploy-sourcegraph-docker#1449 (Docker avoided it only because the extra build was cancelled). The v8.1.1335 cleanup is done by hand: #976 and #977 are closed, and #978 is the "Update main" PR. Co-authored-by: Warren Gifford <warrenbruceg@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test plan: automated release PR, CI will perform additional checks