Repository navigation
Both halves of #1801 are still live on GCP and AWS: branch-keyed concurrency plus unconditional lock deletion #1806
Description
#1801 was fixed for Azure in #1803. Both halves of that bug are still live on GCP and AWS. Filed separately because the scope of #1803 was Azure and the fix is not a copy-paste.
Half 1: branch-keyed concurrency against environment-keyed state
deploy-aws-lambda.yml:33-35->group: deploy-lambda-${{ github.ref }}deploy-aws-fargate.yml:28-30->group: deploy-fargate-${{ github.ref }}deploy-gcp.yml:27-29->group: deploy-gcp-${{ github.ref }}
Each state key is built from the environment, so two runs on different refs that resolve to the same environment land in different concurrency groups and apply against one state file. Identical to the Azure defect.
Half 2: unconditional destruction of a live lock
Both steps run on if: failure() || cancelled() with no age check and no check that the lock is anyone else's:
deploy-gcp.yml:156-167
LOCK_FILE="gs://${BUCKET}/github-${ENVIRONMENT}/default.tflock"
gsutil rm "${LOCK_FILE}" 2>/dev/null || echo "No lock file found (already clean)"
deploy-aws-lambda.yml:269-280
LOCK_KEY="github-${ENVIRONMENT}/terraform.tfstate.tflock"
aws s3 rm "s3://${BUCKET}/${LOCK_KEY}" 2>/dev/null || echo "No lock file found (already clean)"
Deleting another run's live lock object is exactly as destructive as breaking its lease. The lock mechanism differs from Azure's blob lease; the failure mode does not. Combined with half 1, this reproduces #1801's silent state loss: the second writer removes the first's lock, both proceed, and the later writer's stale read drops the earlier one's entries.
An earlier draft of #1803's description claimed the silent-loss half did not transfer to AWS because S3 uses a lock file rather than a lease. That was wrong and is corrected there.
Why this is not a copy-paste of #1803
deploy-aws-lambda.ymlderives its environment differently (target_environment, and it already handles the reusable-workflowgithub.event_namequirk that Reusable deploy workflows read github.event_name as workflow_call, silently downgrading a release deploy to dev #1805 covers).- AWS Fargate uses a separate state key namespace (
github-fargate-<env>/), so it needs its own group, not the lambda one. cleanup-staging.ymlandrollback.ymlalso write the AWS and GCP states and would need the matching groups, same as the Azure change.- Whether to delete the lock-removal steps or make them genuinely conditional needs the same judgement call made in fix(ci/azure): serialize Terraform state writers on environment, not branch #1803, per backend.
Suggested approach
Follow #1803: move each group to the job that writes state, key it on the same value that builds the state key, apply the identical group to every workflow writing that state, and delete the lock-removal steps so a real collision fails loudly.
Activity
cristim commentedon Aug 12, 2026
Fixed in #1812. Three corrections to this issue's scope, found by enumerating all 16
.ymlfiles in.github/workflows/rather than working from the ones named here:destroy-fargate-dev.ymlis a fourth state writer this issue does not mention, and it carried both halves: no concurrency group at all, and an unconditionalaws s3 rmof the.tflockinsideTerraform Init(:111-114on17a568f42). It writesgithub-fargate-dev/terraform.tfstate, the same objectdeploy-aws-fargate.ymlandcleanup-staging.ymlwrite.deploy-gcp.ymlhad two lock deletions, not one. Besides thefailure() || cancelled()step this issue cites,Terraform Initat:145-150rangsutil rmon the lock object unconditionally, on every run, before init, with no condition at all. Strictly worse than the one named here.deploy-aws-fargate.ymlno longer has half 2. This issue correctly lists it under half 1 only, but for the record: its lock-removal step is already gated oninputs.clear_stale_lock == true(aworkflow_dispatch-only boolean defaulting false) and its failure-path release step was already deleted. That step is retained in fix(ci): serialize AWS and GCP Terraform state writers on environment, not branch #1812 as the documented operator-confirmed recovery path.Net: 10 state-mutating jobs across AWS and GCP were ungrouped, not the 3 workflows listed. All 10 now carry a group keyed on the value that builds their own state key. Four lock-deletion steps removed, one retained with justification.
Also filed #1811:
rollback.yml'srollback-aws-fargatebuilds its backend key from the Lambda namespace (github-<env>/terraform.tfstate), notgithub-fargate-<env>/. #1812 therefore gives that job theaws-tfstate-*group, because the group has to name the object the job actually locks; keying it on the Fargate group would have reproduced this exact bug in a new place. Changing the key is a separate and riskier change.fix(ci): add per-database concurrency groups to the database-migratio…
sec(ci): select ECR repos to destroy by exact name, not substring (#1592
fix(scripts): close two fail-open gaps in the state-namespace guard
sec(ci): select ECR repos to destroy by exact name, not substring (#1592