Skip to content

Reusable deploy workflows read github.event_name as workflow_call, silently downgrading a release deploy to dev #1805

Closed

Description

@cristim

Inside a reusable workflow, github.event_name is the caller's event, never the literal string workflow_call. GitHub documents this:

"When a reusable workflow is triggered by a caller workflow, the github context is always associated with the caller workflow."

deploy-azure.yml's prepare job branches on it:

case "$EVENT_NAME" in
  workflow_dispatch|workflow_call) ENVIRONMENT="$INPUT_ENVIRONMENT" ;;
  *)                               ENVIRONMENT=dev ;;
esac

The workflow_call arm is dead code. The consequence is a silent environment downgrade on one path:

deploy-all.yml triggered by the release event sets environment=prod and passes it to deploy-azure.yml. Inside the called workflow github.event_name is release, so prepare takes the *) arm and returns dev. A release-triggered multi-cloud production deploy would deploy Azure to dev, with no error and a summary that says prod.

The workflow_dispatch path survives by luck: dispatching deploy-all.yml makes the caller's event workflow_dispatch, which the first arm happens to match.

The repo already knows about this

deploy-aws-lambda.yml:110-118 documents the behaviour and handles it, including an explicit release) TARGET_ENVIRONMENT=prod ;; arm. That is the pattern to copy.

Affected

  • deploy-azure.yml:88-91 (prepare)
  • deploy-gcp.yml:75-78
  • deploy-aws-fargate.yml:83-86

deploy-aws-lambda.yml:110-118 is already correct and is the reference implementation.

Because deploy-all.yml fans out to all of them, a release-triggered multi-cloud deploy silently targets dev on three clouds while the summary reports prod. That is the real blast radius, and it is why this is worth fixing rather than just documenting.

Evidence status

Documentation plus the existing in-repo comment. No empirical run: gh run list --workflow deploy-all.yml returns [], so deploy-all.yml has never executed and the release path has never been exercised. That is also why this has not bitten anyone yet.

Not a concurrency bug

Found while reviewing #1801 / #1803. It does not affect that fix: the concurrency group and the state key both derive from the same needs.prepare.outputs.environment, so whatever prepare decides, the group always names the blob that actually gets written. This changes which environment is deployed, never whether two writers of one blob serialize.

Activity

  1. cristim commented on Aug 12, 2026

    @cristim
    MemberAuthor

    Re-triaging p2/medium/this-quarter -> p1/high/this-sprint

    Recording the reason rather than changing labels silently.

    I verified the full path on origin/main and the consequence is larger than the original filing conveys: the documented production deploy mechanism silently deploys to dev and reports prod.

    The chain:

    • deploy-all.yml:14 documents the intent: "Release creation (deploys to prod across all clouds)"
    • deploy-all.yml triggers on release: types: [created] as well as workflow_dispatch
    • on the release path, deploy-all.yml:61 sets environment=prod and passes it to each called workflow (:127, :144, :159, :174)
    • deploy-azure.yml's Determine environment reads EVENT_NAME: ${{ github.event_name }} and branches:
    case "$EVENT_NAME" in
      workflow_dispatch|workflow_call) ENVIRONMENT="$INPUT_ENVIRONMENT" ;;
      *)                               ENVIRONMENT=dev ;;
    esac

    Inside a reusable workflow, github.event_name is the caller's event. On the release path it is release, so the workflow_call arm never matches, $INPUT_ENVIRONMENT (= prod) is discarded, and ENVIRONMENT becomes dev.

    Note the workflow_dispatch path works by luck rather than design: the caller's event there really is workflow_dispatch, which happens to match the arm. So the bug is invisible in exactly the way that keeps it alive - the manual path anyone would test with behaves correctly.

    Why this is severity/high

    • Creating a release does not deploy prod. Every release since this was introduced deployed dev instead.
    • It reports success for prod. deploy-all.yml labels the job with the environment it computed (prod), while the called workflow acted on dev. Operators reading the run see a green prod deploy that never happened.
    • It is silent. No error, no warning, no mismatch surfaced. The only way to notice is to compare the deployed image against the release tag.
    • It plausibly affects users, not just internals: fixes believed shipped to production were not. I have left impact/internal rather than raising it, because whether prod is deployed another way is a question for someone with operational context. If prod is in fact released this way, impact should go up too.

    Not raising to p0: there is a working alternative path (workflow_dispatch with an explicit environment), nothing is exposed or corrupted, and no active incident is attributed to it. p1/this-sprint reflects "the production deploy path is broken and lying about it" without claiming it needs someone woken up.

    Fix note

    Do not simply add release to the case arm. The correct signal for "was I called by another workflow" is not github.event_name at all, since that context always describes the triggering event of the whole run. Prefer branching on whether inputs.environment is set, or make the input required on the workflow_call path so an unset value fails loudly instead of defaulting to dev. Any fix must also cover the sibling deploy workflows (deploy-aws-lambda.yml, deploy-gcp.yml) if they share the pattern.

    Whatever lands, the verification is: trigger the release path and confirm the target environment is prod end to end, not merely that the job label says so. The current bug is precisely a label saying prod over an apply against dev.

    Relationship to #1801

    PR #1803 is unaffected: its concurrency group and the state key are derived from the same value, so they stay in lockstep whatever prepare decides. This bug changes which environment is chosen, not whether the group and the blob agree.

  2. added a commit that references this issue on Aug 12, 2026
    c80634e
  3. cristim commented on Aug 12, 2026

    @cristim
    MemberAuthor

    Correction: nothing was ever mis-deployed. This bug is latent, not historical.

    In my re-triage comment I wrote that "every release since this was introduced deployed dev instead" and that it "reports success for prod". Both statements are wrong, and I am retracting them.

    gh run list --workflow=deploy-all.yml returns []. deploy-all.yml has never executed, on any trigger, ever. No release has gone through this path, so no deployment has ever been misrouted and no run has ever reported a prod deploy that went to dev.

    The defect itself is unchanged and still real by inspection: the resolution logic in the called workflows genuinely would land on dev for a release-triggered run passing environment=prod. What is wrong is my claim that it had done so. I stated a production-impact history that does not exist.

    Also corrected: the live blast radius was two clouds, not three

    deploy-all.yml:82 hardcodes deploy-aws-fargate=false, so a release fans out to Lambda, GCP and Azure only. And deploy-aws-lambda.yml already resolved prod, though for the wrong reason - it hardcoded release) prod rather than honouring the input.

    So the paths that would actually have misrouted are Azure and GCP. Fargate was latently broken and is fixed anyway; Lambda produced the right answer by accident.

    Does this change the priority?

    I am leaving it at p1, but the reasoning has to change, and anyone who disagrees should say so now that the facts are straight.

    The case for p1 is no longer "this has been silently breaking production releases". It is that the documented production deploy path (deploy-all.yml:14, "Release creation (deploys to prod across all clouds)") is booby-trapped: the first time anyone uses it as intended, two clouds silently deploy dev while the run summary says prod. A latent defect on the production deploy path, which fails silently and looks green, is worth fixing before someone reaches for it.

    The case for p2 is that it has zero history, zero user impact to date, and the workflow is evidently not in use.

    I lean p1 on the grounds that the cost of it firing once is high and the trigger is a single click. But that is a judgement about future risk, not a report of past damage, and my earlier comment presented it as the latter.

    Note on required: true

    My earlier verification that environment is declared required: true on workflow_call in all the called workflows was correct but insufficient, and I want that recorded rather than left as an implied guarantee. required: true demands that the key be present, not that the value be non-empty. A caller can still pass an empty string and satisfy the schema.

    So the schema does not close this. The fail-loud branch in PR #1809 does, by refusing when the resolved environment is empty rather than defaulting.

  4. added 3 commits that reference this issue on Aug 12, 2026
    dd19342
    eb49e5a
    3b73caa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions