ci(azure): Deploy to Azure Container Apps has failed on every main run for 3 weeks — no Azure change since 2026-07-19 is deployed #1794

Activity

1. cristim

   added

   triagedItem has been triagedItem has been triaged

   priority/p0Drop everything; same-day fixDrop everything; same-day fix

   severity/criticalMajor harm when it happensMajor harm when it happens

   urgency/nowDrop other thingsDrop other things

   impact/all-usersAffects every userAffects every user

   effort/mDaysDays

   type/bugDefectDefect

   on Aug 9, 2026
2. cristim

   added 3 commits that reference this issue on Aug 10, 2026

   fix(ci/azure): cover a plan-time role failure and grep each log separ…

   ...

   e11d47b

   docs(ci/azure): correct two false claims in the diagnostic's comments

   ...

   eb0a872

   fix(ci/azure): name the missing bootstrap stack when the role lookup …

   ...

   ea0578c
3. cristim commented on Aug 11, 2026

   

   cristim

   on Aug 11, 2026

   MemberAuthor

   More actions

   Corrected root cause: the bootstrap role is unappliable, not unapplied

   This issue currently says the custom role was never created because the bootstrap step was missed. That is wrong, and the real cause has a wider blast radius. Established by execution against the live subscription, not by reading.

   The actual cause

   Microsoft.Capacity/reservationOrders/purchase/action does not exist in Azure's operation catalog.

   Applying terraform/environments/azure/ci-cd-permissions fails with:

   Error: unexpected status 400 (400 Bad Request) with error: InvalidActionOrNotAction:
   'Microsoft.Capacity/reservationOrders/purchase/action' does not match any of the
   actions supported by the providers.
   
     with module.cudly_reservation_role.azurerm_role_definition.cudly_reservation_purchaser,
     on ../../../modules/iam/azure/cudly-reservation-role/main.tf line 35
   

   Azure rejects the whole role definition over that one entry, so the role is never created, so the deploy stack's name-based data.azurerm_role_definition lookup finds nothing. The bootstrap stack was not skipped: it cannot succeed as written, and re-running it would never have helped.

   I validated all 12 actions in the module against the live catalog (az provider operation show, covering both namespace-level operations[] and resourceTypes[].operations[]). Exactly one is invalid; the other 11 are fine.

   Hypothesis tested and ruled out

   Microsoft.Capacity and Microsoft.BillingBenefits were both NotRegistered in subscription 24d185cc. That looked like the answer, since Azure validates role actions against registered providers. I registered both and re-applied: identical error. Registration was not the cause.

   The two registrations are additive, reversible, and arguably correct to keep, but they were an incidental change rather than the fix, and I am flagging them rather than leaving them undisclosed.

   The module's stated premise is also wrong

   The module justifies its own existence with:

   The built-in Reservation Purchaser role lacks reservationOrders/purchase/action, which causes 403 on production reservation purchases.

   The built-in role's actual actions do not include Microsoft.Capacity/reservationOrders/write, Microsoft.Capacity/calculatePrice/action, or any Microsoft.BillingBenefits action. So the custom role's purpose is legitimate, but the action added to serve that purpose is fictional. Purchasing a reservation is PUT /providers/Microsoft.Capacity/reservationOrders/{id}, which is reservationOrders/write and was already in the list.

   Blast radius is wider than this issue

   arm/CUDly-CrossSubscription/template.json:51 carries the same invalid action. That template is customer-facing: customers deploy it to grant CUDly cross-subscription access. It should fail identically for every customer who has run it. That deserves its own triage.

   iac/federation/azure-target/terraform is the other customer-side consumer.

   Why no test caught this

   scripts/testdata/role-parity/matching-arm.json and matching-tf.tf.fixture both contain the invalid action. The ARM-vs-TF parity checker therefore passed while both sides were invalid. The fixtures prove the two representations agree; nothing proved either is a thing Azure accepts. Parity is not validity.

   Verification

   Removing the single invalid action and applying against the live subscription created the role:

   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
   reservation_role_definition_id = "/subscriptions/24d185cc-.../roleDefinitions/ab9419be-..."
   

   Confirmed in both directions: fails with the action present, succeeds with it removed.

   Answering the open question about deployed state

   The container app is running cudly:ddeeb98-20260811004918, and ddeeb98 is the current main HEAD. The deploy job builds and ships the image, then fails later during terraform apply when the role lookup is resolved (plan defers it: the log shows data.azurerm_role_definition.cudly_reservation_purchaser will be read during apply).

   So application code from the merged Azure money-path work is live. What has not been applying is the Terraform-managed infrastructure ordered after the failure point. That is a meaningfully better position than the worst case, but the two should not be conflated.

   A rerun of the failed deploy is in flight now that the role exists; I will post the result.

   Note on the CI diagnostic from #1795

   The Explain a missing bootstrap role step added in #1795 tells operators the bootstrap stack was never applied. Given the above, that message points at the wrong cause and is being corrected in the fix PR.
4. cristim

   closed this as completedin #1800on Aug 11, 2026