Skip to content

ci: serialize docs deploy job to prevent gh-pages push races - #2878

Closed
mdboom wants to merge 1 commit into
NVIDIA:mainfrom
mdboom:docs-deploy
Closed

mdboom wants to merge 1 commit into
NVIDIA:mainfrom
mdboom:docs-deploy

Conversation

@mdboom

@mdboom mdboom commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Problem

The build-docs.yml deploy step intermittently fails, and re-running the workflow with no other changes usually makes it pass (e.g. https://github.com/NVIDIA/cuda-python/actions/runs/35119260160/job/104877138975, which failed on the Deploy or clean up doc preview step).

That step pushes rendered docs into the shared gh-pages branch via JamesIves/github-pages-deploy-action, either as a PR preview (docs/pr-preview/pr-<N>/) or as the main/release deploy. The action fetches gh-pages, commits on top, and pushes, with only a small built-in retry.

The existing concurrency group in ci.yml is scoped per-ref (${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}), so it only cancels/serializes runs for the same PR or branch. It does nothing to stop two different refs — e.g. two separate PRs, or a PR preview deploy racing a main deploy — from running their docs-deploy jobs at the same time. When that happens, both jobs push against the same base commit on gh-pages; one wins, the other's push is non-fast-forward, the action's retry budget gets exhausted under contention, and the step fails. Re-running later succeeds once the branch is free, which matches the observed "flaky, but a re-run fixes it" pattern.

Fix

Add a job-level concurrency group on the build job in build-docs.yml, scoped to the shared deploy target rather than the ref:

concurrency:
  group: gh-pages-deploy
  cancel-in-progress: false

Because the group key (gh-pages-deploy) is the same for every run of this job regardless of which ref triggered it, GitHub Actions queues runs that reach this job instead of letting them execute concurrently — this serializes all pushes to gh-pages across PRs, main, and releases. cancel-in-progress: false is intentional: canceling a queued run mid-push could leave gh-pages partially updated, so runs wait their turn rather than being killed.

Test plan

  • Confirm this PR's own build-docs run completes and deploys its preview successfully.
  • Watch subsequent main pushes / PR docs builds for the previously-observed non-fast-forward failure on the deploy step; it should no longer occur.

Concurrent build-docs runs (different PRs, or a PR preview deploy
racing a main/release deploy) all push into the same gh-pages branch.
The existing concurrency group in ci.yml is scoped per-ref, so it does
not prevent two different refs from deploying docs at the same time.
When that happens, the losing push is non-fast-forward and the
JamesIves/github-pages-deploy-action step fails; a re-run usually
succeeds once the branch is free.

Add a job-level concurrency group scoped to the shared deploy target
(gh-pages) so these runs queue instead of racing, with
cancel-in-progress: false to avoid canceling a run mid-push and
leaving gh-pages partially updated.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the CI/CD CI/CD infrastructure label Sep 16, 2026
@mdboom mdboom self-assigned this Sep 16, 2026
@mdboom mdboom added this to the cuda.core 2.0.0 milestone Sep 16, 2026
@mdboom
mdboom requested a review from rwgk September 16, 2026 19:14
@github-actions

Copy link
Copy Markdown
Contributor

@mdboom

mdboom commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favor of #2883, though this can be a backup if that doesn't work.

@mdboom mdboom closed this Sep 18, 2026
github-actions Bot pushed a commit that referenced this pull request Sep 19, 2026
Removed preview folders for the following PRs:
- PR #2878
- PR #2912
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant