Skip to content

fix(iac/aws): rollback environments are absent from the OIDC trust allowlist, so AWS/Azure rollback cannot authenticate #139

Closed
#367
Closed
fix(iac/aws): rollback environments are absent from the OIDC trust allowlist, so AWS/Azure rollback cannot authenticate#139
#367
Labels
effort/sHoursimpact/internalTeam-internal onlypriority/p1Next up; this sprintseverity/highSignificant harmtriagedItem has been triagedtype/bugDefecturgency/this-sprintWithin the current sprint

Description

@cristim
cristim
opened on Jul 28, 2026

Found while fixing LeanerCloud/cloud-commitments-cli#1542 (PR LeanerCloud/cloud-commitments-cli#1641). This is a pre-existing broken deploy path, not a security hole, and not a regression introduced by LeanerCloud/cloud-commitments-cli#1641.

What

.github/workflows/rollback.yml binds its four rollback jobs to environments named:

aws-lambda-<env>-rollback
aws-fargate-<env>-rollback
gcp-<env>-rollback
azure-<env>-rollback

A job with an environment: binding gets the OIDC subject repo:<org/repo>:environment:<name>. But the AWS trust policy allowlist at terraform/environments/aws/ci-cd-permissions/role.tf:28-33 is exactly:

"repo:${var.github_repo}:ref:refs/heads/main",
"repo:${var.github_repo}:environment:dev",
"repo:${var.github_repo}:environment:staging",
"repo:${var.github_repo}:environment:prod",

environment:aws-lambda-prod-rollback is not in that list, so configure-aws-credentials in the rollback job fails with Not authorized to perform sts:AssumeRoleWithWebIdentity. The Azure federated credentials have the same shape (only ref:refs/heads/main and pull_request), so azure/login fails there too.

GCP is unaffected — terraform/environments/gcp/ci-cd-permissions/github_oidc.tf conditions on assertion.repository / assertion.ref and binds by attribute.repository, so the environment name is irrelevant.

Why it is pre-existing

The environment bindings predate LeanerCloud/cloud-commitments-cli#1641 — git show 3e9660d06:.github/workflows/rollback.yml shows rollback-aws-lambda already carrying environment: name: aws-lambda-${{ inputs.environment }}-rollback. So the terraform apply job could never authenticate, before or after.

What LeanerCloud/cloud-commitments-cli#1641 changes is when the operator finds out. Previously the ungated verify-image job authenticated successfully (its subject was ref:refs/heads/main, which the policy does accept) and the failure surfaced at the apply step. LeanerCloud/cloud-commitments-cli#1641 deletes verify-image — that ungated-but-credentialed job was the vulnerability — so the failure now surfaces after the approval prompt instead of before it. Both fail; neither ever completed a rollback.

Fix options

  1. Add the rollback environments to the allowlist — extend role.tf's sub list with repo:<org/repo>:environment:{aws-lambda,aws-fargate,azure}-{dev,staging,prod}-rollback and add matching azuread_application_federated_identity_credential subjects. Keeps a distinct reviewer gate for rollbacks, at the cost of 12 more subjects and 12 more environments to provision.
  2. Reuse the existing environments — rename the workflow's bindings to plain dev / staging / prod. One-line change, no IaC edit, but rollbacks then share the deploy environments' reviewer set rather than having their own.

Option 2 is the smaller change; option 1 is right if rollback is meant to have a different approver list than a normal deploy. That is a policy call, not a technical one.

Also worth noting

GitHub auto-creates a referenced environment on first use with no protection rules. Nothing in this repo provisions these environments or their required reviewers (no github_repository_environment resource anywhere under terraform/ or iac/). So even once the trust policy accepts the subject, the "reviewer gate" is only real if someone configures it in repo settings. Whichever option is chosen should include provisioning the environments and their reviewers, ideally in IaC so it is reviewable.

Note terraform/environments/*/ci-cd-permissions/ is bootstrap-only per CLAUDE.md — applied manually by a privileged human, not by the deploy workflow.

Related: LeanerCloud/cloud-commitments-cli#1591 (environment bindings on destructive workflows), #90 (Azure federated credential trusts pull_request), LeanerCloud/cloud-commitments-cli#1542 / LeanerCloud/cloud-commitments-cli#1641.

Activity

  1. cristim
    added
    triagedItem has been triaged
    priority/p1Next up; this sprint
    severity/highSignificant harm
    urgency/this-sprintWithin the current sprint
    impact/internalTeam-internal only
    effort/sHours
    type/bugDefect
    on Jul 28, 2026

  2. cristim commented on Jul 28, 2026

    @cristim
    cristim
    on Jul 28, 2026
    MemberAuthor

    Empirically confirmed, and worse than this issue assumed. Queried the live API while working PR LeanerCloud/cloud-commitments-cli#1657:

    $ gh api repos/LeanerCloud/CUDly/environments
aws-fargate-dev:     protection_rules=NONE
aws-fargate-staging: protection_rules=NONE
azure-:              protection_rules=NONE
dev:                 protection_rules=NONE
gcp-:                protection_rules=NONE
(total_count: 5)

    Three things follow:

    1. No Environment in this repo has any protection rules. So environment: today only scopes secrets/vars and sets the OIDC sub claim — it is not a reviewer gate anywhere, in any workflow. Every workflow that relies on an environment binding as an approval control (rollback.yml, database-migration.yml, the deploy-* workflows) is currently ungated in practice.
    2. staging and prod do not exist as Environments at all, only dev. Yet the AWS trust policy allowlists environment:dev, environment:staging and environment:prod. So the subjects the policy accepts and the environments that actually exist are already out of sync, independently of the <cloud>-<env>-rollback gap this issue was filed about.
    3. New, unrelated bug: azure- and gcp- are junk environments created by an environment: name interpolating to an empty string (i.e. azure-${{ ... }} where the expression was empty). GitHub auto-creates whatever name it is given, so these are the fossil record of a workflow run with an unset environment input. Worth tracking down which workflow produced them — an environment name that silently resolves to azure- also means the OIDC subject was repo:<org/repo>:environment:azure-, which the trust policy would reject, so that run's auth would have failed.

    This strengthens the case for provisioning the environments and their required reviewers in IaC (github_repository_environment + required_reviewers) rather than relying on auto-creation, since auto-creation is what produced both the missing protection rules and the junk names.

  3. cristim commented on Jul 28, 2026

    @cristim
    cristim
    on Jul 28, 2026
    MemberAuthor

    Azure has the same gap — this issue is not AWS-only. Verified on current main:

    terraform/environments/azure/ci-cd-permissions/sp.tf carries exactly two federated-credential subjects:

    :37  subject = "repo:${var.github_repo}:ref:refs/heads/main"
:48  subject = "repo:${var.github_repo}:pull_request"

    No environment:* subject at all — not even environment:dev/staging/prod, which AWS at least has. So azure/login in rollback-azure (subject repo:<org/repo>:environment:azure-<env>-rollback) fails with AADSTS70021, exactly as the AWS path does. Azure is arguably worse than AWS here: AWS allowlists the three plain environment names, Azure allowlists none.

    Suggest retitling this issue to cover both clouds rather than splitting it, since the fix is the same shape in two files and they must land together for the rollback path to work end to end:

    • AWS: add the rollback subjects to the sub list in role.tf:28-33.
    • Azure: add matching azuread_application_federated_identity_credential resources in sp.tf (Azure needs one resource per subject — no wildcards — so this is 12 resources if the per-cloud-per-env rollback naming is kept, which is an argument for option 2 in the original description: rename the workflow's bindings to plain dev/staging/prod and reuse the existing subjects).

    GCP remains unaffected for authentication purposes — github_oidc.tf:37 conditions on assertion.repository and assertion.ref only, so the environment name is irrelevant to whether the token mints. That permissiveness is its own problem and is filed as part of #141.

    (Also note sp.tf:48's pull_request subject is #90's concern, untouched here.)

  4. cristim
    added 3 commits that reference this issue on Jul 29, 2026

    fix(ci): correct the destroy workflows' claims about what the ref che…

    c39adc3

    fix(iac/aws): extend OIDC trust allowlist to cover every job assuming…

    a826688

    sec(ci): bind destroy jobs to an environment and narrow id-token on d…

    f6bb36a
  5. cristim
    transferred this issue fromLeanerCloud/cloud-commitments-clion Sep 27, 2026
  6. cristim
    added 2 commits that reference this issue on Sep 28, 2026

    fix(ci): bind rollback.yml jobs to dev/staging/prod, not <cloud>-<env…

    4c43f68

    docs(ci): regenerate the environments list from actual environment: b…

    5a1749f
  7. cristim
    closed this as completedin #367on Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/sHoursimpact/internalTeam-internal onlypriority/p1Next up; this sprintseverity/highSignificant harmtriagedItem has been triagedtype/bugDefecturgency/this-sprintWithin the current sprint

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions