feat: Release Tracks — floating Docker tags (latest/standard/trailing) promotion engine (#36160)

[sfreudenthaler]

Resolves #36160 · Epic #35693

Proposed Changes

• Add a small Python promotion engine at cicd/evergreen-tracks/ (managed with uv) that advances three floating Docker tags — latest / standard / trailing — across the linear GA CalVer stream by release age (newest GA, ~14d, ~28d; thresholds configurable).
• State is registry-only via marker tags: <version>_tainted (forward-only block — a bad release can't propagate to more conservative tracks) and <track>_hold (sticky manual freeze). No separate datastore; the audit trail is the Actions run logs.
• Tags are re-pointed by digest with docker buildx imagetools create (no layer re-push). Age is read from the CalVer date in the version, not build/publish date, so emergency backports of older releases can't be swept into a future promotion.
• Two workflows: cicd_evergreen-tracks-promote.yml (daily cron + dispatch) and cicd_evergreen-tracks-admin.yml (manual taint/hold).
• Document Release Tracks (usage + the "why") in the root README.md.

Checklist

• Tests — 60 unit tests (pure planner, calver, markers, registry parser, CLI); run with cd cicd/evergreen-tracks && uv run pytest.
• Translations — N/A (CI tooling, no UI strings).
• Security Implications Contemplated — see notes below.

Additional Info

Tag control: latest is moved on-demand by the release pipeline (promote-latest job in cicd_6-release.yml, --tracks latest) the moment a GA's images publish, for dotcms/dotcms and dotcms/dotcms-dev — the old deploy-docker latest: true path is unwired (latest: false). The daily cron ages standard/trailing forward and always applies (no separate enable gate). To pause promotion, disable the scheduled workflow or hold the track; to block a bad release, taint it. All registry mutations (release-driven latest, cron, admin) serialize under one concurrency group evergreen-tracks-registry.

Credential scope (important): promotion needs only write, but untaint and release-hold call the Hub delete API — the DOCKER_USERNAME/DOCKER_TOKEN used here must have Read/Write/Delete scope, or those two admin actions fail. (Verified both ways: write-only 403s on delete; RWD succeeds.)

Validation: the full lifecycle (promote-by-age, taint→skip, hold→freeze, release-hold→resume, untaint→restore, teardown) was exercised end-to-end against the dotcms/dotcms-test sandbox and verified by digest. The live smoke can't run in core-workflow-test CI (it intentionally carries no live Docker secrets), so it should run here in core CI / on dispatch.

Security notes: free-text workflow inputs are passed via env: (not interpolated into run:) to avoid expression injection; no tokens are logged; least-privilege permissions: contents: read on the promote workflow.

Out of scope (per epic): LTS-line tracks, Java-variant track tags, the Cloud control-plane UI, and update cadence changes.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[claude]

Claude finished @sfreudenthaler's task in 1m 3s —— View job

---

Rollback Safety Analysis

• Gather context and understand request
• Read rollback-unsafe categories reference
• Get full PR diff
• Analyze changes against each unsafe category
• Apply label

---

Result: ✅ Safe To Rollback

All 20 changed files are pure additions (+1659 / -0) in three buckets:

Bucket	Files	Rollback Impact
CI/CD workflows	.github/workflows/cicd_evergreen-tracks-admin.yml, cicd_evergreen-tracks-promote.yml	No database, ES, or API changes. Workflows only run on explicit trigger or cron after EVERGREEN_TRACKS_APPLY=true is set.
Python promotion engine	cicd/evergreen-tracks/ (source, tests, pyproject.toml, uv.lock)	Self-contained tooling in a subdirectory; no Java, no DB schema, no ES mapping, no REST/GraphQL surface.
Documentation	README.md, cicd/evergreen-tracks/README.md	Docs only.

Category-by-category verdict:

Category	Match?
C-1 Structural Data Model Change	❌ No DB schema changes
C-2 Elasticsearch Mapping Change	❌ No ES mapping touched
C-3 Content JSON Model Version Bump	❌ No contentlet_as_json changes
C-4 DROP TABLE / DROP COLUMN	❌ No DDL drops
H-1 One-Way Data Migration	❌ No data transformation tasks
H-2 RENAME TABLE / RENAME COLUMN	❌ None
H-3 PK / Unique Constraint Restructuring	❌ None
H-4 New Field Type	❌ None
H-5 Storage Provider Change	❌ None
H-6 DROP PROCEDURE / FUNCTION	❌ None
H-7 NOT NULL Column Without Default	❌ None
H-8 VTL Viewtool Contract Change	❌ None
M-1 Column Type Change	❌ None
M-2 Push Publishing Bundle Format	❌ None
M-3 REST / GraphQL API Contract	❌ None
M-4 OSGi Interface Change	❌ None

Rolling back dotCMS from a release that included this PR to the previous release is fully safe: the promotion engine lives entirely in CI/CD and leaves no persistent state in the database, Elasticsearch, or any other shared datastore. The only external side-effect is Docker Hub tag mutations, which are controlled by the separate EVERGREEN_TRACKS_APPLY gate and have their own marker-tag-based audit trail in the registry.

Label added: AI: Safe To Rollback

[github-actions]

🤖 Codex Review — openai.gpt-5.5

**[> [🟠 High] .github/workflows/cicd_evergreen-tracks-admin.yml:34 — DOCKER_USERNAME / DOCKER_TOKEN are set at job scope, so every step/action in the job receives Docker Hub credentials, including third-party setup actions that don’t need them. Keep these secrets scoped only to docker/login-action and the final admin run step.

**[> [🟠 High] .github/workflows/cicd_evergreen-tracks-admin.yml:34 — DOCKER_USERNAME / DOCKER_TOKEN are set at job scope, so every step/action in the job receives Docker Hub credentials, including third-party setup actions that don’t need them. Keep these secrets scoped only to docker/login-action and the final admin run step.

[🟠 High] .github/workflows/cicd_evergreen-tracks-promote.yml:34 — the concurrency group is scoped to ${{ github.workflow }} and ${{ github.ref }}, while the admin workflow has no shared concurrency group. Promote runs from different refs, and promote/admin runs, can still overlap while reading and mutating the same registry tags, which can overwrite holds/taints or move tracks from stale state.

**[> [🟠 High] .github/workflows/cicd_evergreen-tracks-admin.yml:34 — DOCKER_USERNAME / DOCKER_TOKEN are set at job scope, so every step/action in the job receives Docker Hub credentials, including third-party setup actions that don’t need them. Keep these secrets scoped only to docker/login-action and the final admin run step.

[🟠 High] .github/workflows/cicd_evergreen-tracks-promote.yml:34 — the concurrency group is scoped to ${{ github.workflow }} and ${{ github.ref }}, while the admin workflow has no shared concurrency group. Promote runs from different refs, and promote/admin runs, can still overlap while reading and mutating the same registry tags, which can overwrite holds/taints or move tracks from stale state.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:89 — untaint calls hub_login() before delete_tag(..., apply=args.apply), so even dry-runs require Docker credentials and make a live Docker Hub login request. This breaks the documented dry-run behavior and prevents credential-free validation.

**[> [🟠 High] .github/workflows/cicd_evergreen-tracks-admin.yml:34 — DOCKER_USERNAME / DOCKER_TOKEN are set at job scope, so every step/action in the job receives Docker Hub credentials, including third-party setup actions that don’t need them. Keep these secrets scoped only to docker/login-action and the final admin run step.

[🟠 High] .github/workflows/cicd_evergreen-tracks-promote.yml:34 — the concurrency group is scoped to ${{ github.workflow }} and ${{ github.ref }}, while the admin workflow has no shared concurrency group. Promote runs from different refs, and promote/admin runs, can still overlap while reading and mutating the same registry tags, which can overwrite holds/taints or move tracks from stale state.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:89 — untaint calls hub_login() before delete_tag(..., apply=args.apply), so even dry-runs require Docker credentials and make a live Docker Hub login request. This breaks the documented dry-run behavior and prevents credential-free validation.

[🟡 Medium] cicd/evergreen-tracks/src/evergreen_tracks/cli.py:114 — release-hold has the same dry-run issue: it logs into Docker Hub before checking/applying the delete, so --apply false still requires credentials and external network access.

---

_{Run: #27556377640 · tokens: in: 26182 · out: 7564 (reasoning: 7230) · total: 33746}

[semgrep-code-dotcms-test]

Legal Risk

certifi 2026.5.20 was released under the MPL-2.0 license, a license that
has been flagged by your organization for consideration.

Recommendation

While merging is not directly blocked, it's best to pause and consider what it means to use this license before continuing. If you are unsure, reach out to your security team or Semgrep admin to address this issue.

[sfreudenthaler]

🟠 High — .github/workflows/cicd_evergreen-tracks-promote.yml:27: the promote workflow has no concurrency group, so a scheduled run and a manual promote/admin run can overlap, both read registry state, then apply stale tag moves.

Addressed in 15f171a. Added a workflow-level concurrency group keyed by github.workflow + github.ref with cancel-in-progress: false, so a scheduled run and a manual promote/admin dispatch are serialized — concurrent runs queue and wait their turn rather than aborting mid-promote and leaving tags half-moved. Queueing (not cancelling) is the safer choice for a tag-promotion engine.

[wezell]

Wait, so :latest is going to become 2 weeks old? Should we maintain latest as is and add another tag called... :current? latest is a well known convention.

[sfreudenthaler]

Wait, so :latest is going to become 2 weeks old? Should we maintain latest as is and add another tag called... :current? latest is a well known convention.

after live discussion decision is to hook our releaser into this as a on-demand invoke that way

• latest moves right away
• only one tool asserts control moving evergreen track tags

[sfreudenthaler]

Wait, so :latest is going to become 2 weeks old? Should we maintain latest as is and add another tag called... :current? latest is a well known convention.

Good catch — latest stays exactly as you'd expect (newest GA), and we keep the convention. Per our live discussion, I wired the releaser into this engine so there's one controller for all three tags instead of two. Pushed in 3539720:

• latest moves on-demand, immediately. A new promote-latest job in cicd_6-release.yml invokes the evergreen-tracks engine right after the release images publish (--tracks latest --apply), for both dotcms/dotcms and dotcms/dotcms-dev. No 24h cron lag.
• Unwired the old path. cicd_6-release.yml no longer asks deploy-docker to move latest (latest: false). The engine is now the single mover of latest/standard/trailing.
• Rollout safety preserved. The release-triggered latest promote applies unconditionally (it replaces always-on behavior); it is not behind EVERGREEN_TRACKS_APPLY, which still gates only the new aged standard/trailing tracks until we flip it on.
• Race closed. All registry mutations (daily cron, admin, and this release-triggered promote) now serialize under one static concurrency group evergreen-tracks-registry — this also addresses the remaining cross-workflow concurrency point from the Codex review.

So standard ≈ 2 weeks and trailing ≈ 4 weeks are the aged tracks; latest is unchanged.