Repository navigation
Reservation-purchaser role assignment is destroyed and recreated on every apply, briefly revoking a money-path grant #1802
Description
module.compute_container_apps[0].azurerm_role_assignment.reservations_purchaser is destroyed and recreated on every single apply, so the host identity's reservation-purchase grant is briefly revoked on every deploy.
Found while verifying the #1794 fix in deploy run 31545175691, where it appears as one of exactly three destroys.
Why
Both role_definition_id and scope derive from a data source that is deferred to apply time (data.azurerm_role_definition.cudly_reservation_purchaser, which Terraform reports as will be read during apply). Both attributes are therefore always (known after apply), and both force replacement on azurerm_role_assignment. So the resource can never be seen as unchanged, regardless of whether anything actually changed.
Why it matters
This is a money-path grant. It is what lets the host identity call calculatePrice and reservationOrders/write. There is a window on every deploy where the assignment does not exist, and a purchase attempting to run in that window would 403.
The window is short, but the deploy is not coordinated with purchase activity in any way, so nothing rules the overlap out. It also produces misleading churn in every plan, which trains reviewers to skim past a destroy line on a permissions resource, and that is its own hazard.
Fix direction
Stop making the assignment depend on apply-time-unknown values. Options worth weighing:
- Resolve the role definition by a stable, statically-known ID rather than a name lookup, so
role_definition_idis known at plan time. The bootstrap stack already outputsrole_definition_resource_id; consuming that through a documented channel (remote state or an explicit variable) removes the data source entirely. - Failing that, pin
scopeto a statically-derivable subscription ID rather than routing it through the deferred data source.
Note the deliberate bootstrap-vs-runtime split documented in terraform/modules/compute/azure/container-apps/main.tf: the role definition is created by the human-applied bootstrap stack, and the runtime stack only looks it up. Any fix has to respect that split rather than move definition ownership into the runtime stack.
Verification
Apply twice with no intervening changes and confirm the second plan reports no changes for this resource. That is the whole test, and today it fails.
Then confirm the grant is continuously present across a deploy rather than merely present afterwards, since an assignment that is destroyed and recreated still ends in the correct final state while having been absent in between.
Related
- ci(azure): Deploy to Azure Container Apps has failed on every main run for 3 weeks — no Azure change since 2026-07-19 is deployed #1794 / PR fix(iac/azure): remove nonexistent Microsoft.Capacity action from reservation-purchaser role #1800 - the role-definition fix that made this visible, by letting apply reach this resource at all.
- Azure deploy silently loses Terraform state: branch-keyed concurrency plus an unconditional lease break #1801 - separate state-integrity defect in the same pipeline.
Activity
cristim commentedon Aug 12, 2026
Correction: this affects all three subscription-scoped role assignments, not just the purchaser
I filed this naming only
reservations_purchaser. That was too narrow, and the cause I gave was a symptom rather than the root.Verified against the apply log of run 31545175691 (the first green Azure deploy since 2026-07-19):
All three are destroyed and recreated on every apply.
Actual root cause
module "compute_container_apps"carries a module-leveldepends_on(terraform/environments/azure/compute.tf:75) that targetsmodule.buildamong others, andmodule.buildchanges on every deploy. Terraform propagates a module-leveldepends_onto every data source in the module, so both module data sources are deferred to apply time. That makesscope(fromdata.azurerm_subscription.current)(known after apply)for all three assignments, androle_definition_idadditionally unknown for the purchaser.So the purchaser is replaced on two unknown attributes and the other two on one. My original diagnosis identified only the second of those, which is why I named only one resource.
Why the wider scope matters
The revocation window on each deploy covers:
calculatePriceandreservationOrders/write, the money pathThat is not one grant briefly flickering, it is the app's entire host-subscription read and purchase capability. Anything running during a deploy can 403 on any of the three.
I am leaving the priority at p2 rather than raising it unilaterally, because the window is short and no incident is attributed to it. Anyone who disagrees given the wider blast radius should re-triage it explicitly rather than quietly working around the label.
Note on the fix
PR #1804 resolves both inputs in the caller, so it covers all three assignments rather than only the purchaser. It deliberately does not delete the module
depends_on, which is the true root cause and a one-line change, because thatdepends_onalso orders the container app after resources no referenced output covers (NSG associations, jwt/session secrets, DB firewall rules). Removing it changes greenfield-apply ordering, which is a larger change than this issue. That tradeoff is recorded in the PR.Still unverified
The real test is two consecutive live applies with no intervening change, with the second plan reporting no changes for these resources. That has not run.
terraform validatepassing is not evidence for this.fix(iac/azure): fail at plan time when subscription_id is not a GUID
fix(iac/azure): stop replacing the reservation-purchaser role assignm…
cristim commentedon Aug 12, 2026
Verified in production: the churn has stopped
PR #1804 shipped with this explicitly unverified, because the real test needs two consecutive live applies and could not be run from a developer machine. Those applies have now happened.
Before the fix - run
31545175691(head51e89a6, pre-#1804):After the fix - two consecutive successful deploys, both with
module.compute_container_appsin scope:315740488622c87db0315764834378b16ed4Every one of the ten role assignments appears only as
Refreshing state.... None is destroyed, created or modified in either run. The money-path grant (calculatePrice+reservationOrders/write) is no longer revoked on each deploy, and neither areCost Management ReaderorReader.Limits of this evidence, stated plainly
gh run view --logappears to truncate: neither captured log contains anApply complete!line or aPlan:summary, so this is not a plan-summary-level proof of "no changes". What it is: two post-fix applies in which none of the ten assignments shows a mutation verb, against a pre-fix apply in which three of them showDestroyingandCreatingexplicitly.That is a real before/after signal on the same log source and the same extraction, and both runs concluded
successacross all four jobs. It is strong indicative evidence rather than the definitive plan diff. Anyone wanting the stricter proof should read theTerraform Planstep output of a future run directly and confirm it reports no changes formodule.compute_container_apps[0].azurerm_role_assignment.reservations_purchaser.One prediction from the PR that held
PR #1804 noted that if the
AZURE_SUBSCRIPTION_IDsecret's casing differed from Azure's canonical lowercase, the first plan after the change would still show one final replacement before converging, and thatlower()is what makes it converge rather than churn forever. Two consecutive clean runs are consistent with convergence having already happened.