Skip to content

Commit 304096e

Browse files
Thomas-BoyleThomas-Boyle
authored
VED-1223: Update permissions to auto-ops role so the pipeline can apply terraform changes at account level (#1384)
* Add account-level Terraform workflow and integrate with existing CI/CD pipelines - Introduced a new GitHub Actions workflow for managing account-level Terraform operations, including planning, manual approval, and applying changes. - Updated `continuous-deployment.yml` and `pr-deploy-and-test.yml` to utilize the new account-terraform workflow, ensuring infrastructure changes are detected and processed. - Enhanced Makefile with new targets for CI-specific Terraform commands. - Added a script to resolve the Terraform state bucket dynamically based on the configured environment. - Updated IAM policy to include permissions for AWS Shield operations. * Add bucket name validation to Makefile and improve bucket resolution script - Introduced a new function in the Makefile to ensure the BUCKET_NAME variable is set before executing Terraform commands. - Updated the bucket resolution script to trim whitespace from the CONFIGURED_ACCOUNT_TERRAFORM_STATE_BUCKET variable, enhancing reliability in detecting the configured bucket. * Enhance bucket name resolution with validation in account-terraform workflow - Added validation to ensure the resolved bucket name is not empty, improving error handling in the GitHub Actions workflow. - Updated the script execution to store the bucket name in a variable before outputting, enhancing clarity and maintainability. * Add state bucket environment input to workflows and update bucket resolution script - Introduced a new optional input `state_bucket_environment` in the `account-terraform.yml` workflow to allow dynamic configuration. - Updated `continuous-deployment.yml` and `pr-deploy-and-test.yml` workflows to set the `state_bucket_environment` for internal development. - Enhanced the bucket resolution script to utilize the `ACCOUNT_TERRAFORM_STATE_ENVIRONMENT` variable for improved bucket naming based on the environment. * Enhance Terraform workspace handling in workflows and bucket resolution script - Added `ACCOUNT_TERRAFORM_WORKSPACE` input to the `account-terraform.yml` workflow for improved workspace management. - Updated the bucket resolution script to validate the `ACCOUNT_TERRAFORM_WORKSPACE` variable and incorporate it into the state key for better bucket identification. - Enhanced error handling to ensure the workspace is set before proceeding with bucket resolution. * Enhance bucket resolution script and workflow logging - Added a new function to the bucket resolution script to check if the bucket matches the account state, improving accuracy in identifying the correct bucket. - Updated the workflow to log the resolved bucket name, enhancing visibility during execution and debugging. * Refactor bucket resolution script to output bucket name and exit - Updated the `resolve_account_terraform_state_bucket.sh` script to print the formatted bucket name based on the `state_bucket_environment` variable. - Removed the previous logic for adding candidate buckets, simplifying the script's flow when the environment variable is set. * Refactor bucket resolution script to improve whitespace handling and error messaging - Simplified the script by introducing a `trim` function to handle whitespace for the `CONFIGURED_ACCOUNT_TERRAFORM_STATE_BUCKET` and `ACCOUNT_TERRAFORM_STATE_ENVIRONMENT` variables. - Enhanced error handling to ensure the `ACCOUNT_TERRAFORM_STATE_ENVIRONMENT` variable is set when the configured bucket is not provided, improving clarity in user feedback. * Update account-terraform workflow to conditionally set bucket name based on environment - Modified the `CONFIGURED_ACCOUNT_TERRAFORM_STATE_BUCKET` variable to use a conditional expression, allowing for a default bucket name when the environment is 'dev'. - Removed the `ACCOUNT_TERRAFORM_WORKSPACE` variable from the environment settings to streamline the workflow. * Refactor account-terraform workflow to streamline job dependencies and conditions - Consolidated the account-terraform jobs by removing unnecessary manual approval and not-required stages, simplifying the workflow. - Updated conditions for the account-terraform-apply job to ensure it only runs when infrastructure changes are detected. - Enhanced the account-terraform-plan job to skip execution when no changes are present, improving efficiency. * Enhance account-terraform workflow to improve change detection logging - Updated the change detection logic to store and log specific files that have changed within the `infrastructure/account/` directory, enhancing visibility during workflow execution. - Ensured that the workflow outputs a clear indication of whether account infrastructure changes are present, improving debugging and tracking of modifications. * Update pr-deploy-and-test workflow to enhance SHA handling during synchronization - Modified the logic for setting `base_sha` and `diff_base_sha` to account for the 'synchronize' action, ensuring accurate detection of changes when pull requests are updated. - Improved the workflow's responsiveness to changes, enhancing overall efficiency in deployment and testing processes. * Enhance account-terraform workflow to improve environment variable handling - Added environment variables for base SHA, head SHA, and environment to streamline the Terraform job configurations. - Updated the workflow steps to utilize these environment variables, improving clarity and maintainability in the execution of Terraform commands. * Enhance account-terraform workflow with improved SHA validation and environment variable usage - Added validation for base and head SHA variables to ensure they are correctly formatted before proceeding with change detection. - Updated the workflow to utilize an environment variable for the bucket name in Terraform initialization steps, enhancing clarity and maintainability. * Refactor account-terraform workflow and bucket resolution script for improved change detection and whitespace handling - Updated the SHA validation logic in the account-terraform workflow to use more efficient conditional expressions. - Enhanced change detection by directly capturing modified files in the `infrastructure/account/` directory, improving logging clarity. - Refined the `trim` function in the bucket resolution script to handle whitespace more effectively, ensuring accurate bucket name processing. * Refactor account-terraform workflow and bucket resolution script for improved clarity and efficiency - Introduced environment variables for the configured bucket and state environment directly in the workflow, enhancing maintainability. - Simplified the bucket resolution logic in the script by condensing conditional checks, improving readability and error handling. - Streamlined the workflow steps to directly capture the bucket name output, reducing unnecessary complexity. * Refactor account-terraform workflow and bucket resolution script for enhanced validation and clarity - Consolidated SHA validation logic in the account-terraform workflow to improve efficiency and readability. - Streamlined change detection output by directly capturing the status of account infrastructure changes. - Simplified the bucket resolution script by removing unnecessary functions and utilizing direct variable assignment for improved clarity. * Update base_sha logic in pr-deploy-and-test workflow to simplify SHA handling for pull requests. Removed conditional check for 'synchronize' action to ensure consistent base SHA retrieval. * chore: empty commit * chore: empty commit * Enhance account-terraform workflow by adding manual approval step before apply phase - Introduced a new job for manual approval after the plan phase, ensuring that changes are reviewed before application. - Updated the apply job to depend on the approval step, enhancing control over the deployment process. * Fix description in ECR lifecycle policy to include a period for consistency. * Refactor account-terraform workflow to enhance input handling and artifact naming - Updated workflow name for clarity. - Added support for manual input parameters including environment selection and optional artifact naming. - Improved handling of SHA values for better consistency in deployment processes. - Streamlined artifact name generation to ensure clarity and avoid potential conflicts. * Enhance account-terraform workflow and scripts for improved stability and clarity - Added ACCOUNT_TERRAFORM_VERSION environment variable for consistent Terraform versioning. - Increased job timeout to 30 minutes for both planning and applying stages. - Updated role-session-name format for better traceability in AWS actions. - Modified Makefile to streamline Terraform apply command. - Added validation for ACCOUNT_TERRAFORM_STATE_ENVIRONMENT in the state bucket resolution script to enforce correct environment values. * Enhance account-terraform workflow with attestation support - Added permissions for attestations and artifact metadata in the workflow. - Introduced steps for attesting the Terraform plan and verifying the attestation. - Improved overall security and traceability of the Terraform deployment process. * chore: empty commit * chore: empty commit
1 parent a89b0aa commit 304096e

6 files changed

Lines changed: 270 additions & 2 deletions

File tree

‎.github/workflows/account-terraform.yml‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
name: Apply Account Terraform
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
base_sha:
7+
required: true
8+
type: string
9+
head_sha:
10+
required: true
11+
type: string
12+
environment:
13+
required: true
14+
type: string
15+
state_bucket_environment:
16+
required: false
17+
type: string
18+
default: ""
19+
artifact_name:
20+
required: true
21+
type: string
22+
workflow_dispatch:
23+
inputs:
24+
environment:
25+
description: Select AWS account environment
26+
required: true
27+
type: choice
28+
options:
29+
- dev
30+
- preprod
31+
- prod
32+
state_bucket_environment:
33+
description: Override state bucket environment
34+
required: false
35+
type: string
36+
default: ""
37+
base_sha:
38+
description: Base commit SHA for diff checks. Leave blank to use previous commit.
39+
required: false
40+
type: string
41+
default: ""
42+
head_sha:
43+
description: Head commit SHA for diff checks. Leave blank to use current commit.
44+
required: false
45+
type: string
46+
default: ""
47+
artifact_name:
48+
description: Optional Terraform plan artifact name
49+
required: false
50+
type: string
51+
default: ""
52+
53+
run-name: Apply Account Terraform - ${{ inputs.environment }}
54+
55+
concurrency:
56+
group: account-terraform-${{ github.repository }}-${{ inputs.environment }}
57+
cancel-in-progress: false
58+
59+
env:
60+
CONFIGURED_ACCOUNT_TERRAFORM_STATE_BUCKET: ${{ vars.ACCOUNT_TERRAFORM_STATE_BUCKET || (inputs.environment == 'dev' && 'immunisation-terraform-state-files' || '') }}
61+
ACCOUNT_TERRAFORM_STATE_ENVIRONMENT: ${{ inputs.state_bucket_environment }}
62+
ACCOUNT_TERRAFORM_ARTIFACT_NAME: ${{ inputs.artifact_name || format('{0}-account-tfplan-{1}', inputs.environment, github.run_attempt) }}
63+
ACCOUNT_TERRAFORM_VERSION: "1.12.2"
64+
65+
jobs:
66+
account-terraform-plan:
67+
permissions:
68+
id-token: write
69+
contents: read
70+
attestations: write
71+
artifact-metadata: write
72+
runs-on: ubuntu-latest
73+
timeout-minutes: 30
74+
environment:
75+
name: ${{ inputs.environment }}
76+
env:
77+
ACCOUNT_TERRAFORM_BASE_SHA: ${{ inputs.base_sha }}
78+
ACCOUNT_TERRAFORM_HEAD_SHA: ${{ inputs.head_sha || github.sha }}
79+
ACCOUNT_TERRAFORM_ENVIRONMENT: ${{ inputs.environment }}
80+
outputs:
81+
account_infra_changed: ${{ steps.diff.outputs.account_infra_changed }}
82+
plan_sha: ${{ env.ACCOUNT_TERRAFORM_HEAD_SHA }}
83+
steps:
84+
- name: Checkout
85+
uses: actions/checkout@0c366fd6a839edf440554fa01a7085ccba70ac98
86+
with:
87+
fetch-depth: 0
88+
89+
- name: Detect account terraform changes
90+
id: diff
91+
run: |
92+
base_sha="$ACCOUNT_TERRAFORM_BASE_SHA"
93+
head_sha="$ACCOUNT_TERRAFORM_HEAD_SHA"
94+
95+
if [[ -z "$base_sha" || "$base_sha" == "0000000000000000000000000000000000000000" ]]; then
96+
base_sha=$(git rev-parse HEAD~1)
97+
fi
98+
99+
for sha_name in base_sha head_sha; do
100+
if [[ ! "${!sha_name}" =~ ^[0-9a-f]{40}$ ]]; then
101+
echo "Invalid $sha_name: ${!sha_name}" >&2
102+
exit 1
103+
fi
104+
done
105+
106+
account_changed_files=$(git diff --name-only "$base_sha" "$head_sha" -- infrastructure/account)
107+
if [ -n "$account_changed_files" ]; then
108+
echo "changes detected in files:"
109+
printf '%s\n' "$account_changed_files"
110+
fi
111+
echo "account_infra_changed=$( [ -n "$account_changed_files" ] && echo true || echo false )" >> "$GITHUB_OUTPUT"
112+
113+
- name: Connect to AWS
114+
if: ${{ steps.diff.outputs.account_infra_changed == 'true' }}
115+
uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
116+
with:
117+
aws-region: eu-west-2
118+
role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/auto-ops
119+
role-session-name: ${{ format('github-actions-{0}-{1}-{2}', github.run_id, github.run_attempt, github.job) }}
120+
121+
- uses: hashicorp/setup-terraform@5e8dbf3c6d9deaf4193ca7a8fb23f2ac83bb6c85
122+
if: ${{ steps.diff.outputs.account_infra_changed == 'true' }}
123+
with:
124+
terraform_version: ${{ env.ACCOUNT_TERRAFORM_VERSION }}
125+
126+
- name: Resolve account terraform state bucket
127+
id: account-state-bucket
128+
if: ${{ steps.diff.outputs.account_infra_changed == 'true' }}
129+
run: echo "bucket_name=$(bash ./utilities/scripts/resolve_account_terraform_state_bucket.sh)" >> "$GITHUB_OUTPUT"
130+
131+
- name: Terraform Init (account)
132+
if: ${{ steps.diff.outputs.account_infra_changed == 'true' }}
133+
working-directory: infrastructure/account
134+
env:
135+
ACCOUNT_TERRAFORM_BUCKET_NAME: ${{ steps.account-state-bucket.outputs.bucket_name }}
136+
run: make init ENVIRONMENT="$ACCOUNT_TERRAFORM_ENVIRONMENT" BUCKET_NAME="$ACCOUNT_TERRAFORM_BUCKET_NAME"
137+
138+
- name: Terraform Plan (account)
139+
# Ignore cancellations to prevent Terraform from being killed while it holds a state lock
140+
# A stuck process can still be killed with the force-cancel API operation
141+
if: ${{ steps.diff.outputs.account_infra_changed == 'true' && !failure() }}
142+
working-directory: infrastructure/account
143+
run: make plan-ci ENVIRONMENT="$ACCOUNT_TERRAFORM_ENVIRONMENT"
144+
145+
- name: Save Account Terraform Plan
146+
if: ${{ steps.diff.outputs.account_infra_changed == 'true' }}
147+
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f
148+
with:
149+
name: ${{ env.ACCOUNT_TERRAFORM_ARTIFACT_NAME }}
150+
path: infrastructure/account/tfplan
151+
152+
- name: Attest Account Terraform Plan
153+
if: ${{ steps.diff.outputs.account_infra_changed == 'true' }}
154+
uses: actions/attest@v4
155+
with:
156+
subject-path: infrastructure/account/tfplan
157+
158+
account-terraform-approval:
159+
permissions: {}
160+
needs: [account-terraform-plan]
161+
if: ${{ !cancelled() && needs.account-terraform-plan.result == 'success' && needs.account-terraform-plan.outputs.account_infra_changed == 'true' }}
162+
runs-on: ubuntu-latest
163+
environment:
164+
name: account-apply-${{ inputs.environment }}
165+
steps:
166+
- name: Await manual approval
167+
run: echo "Manual approval granted"
168+
169+
account-terraform-apply:
170+
permissions:
171+
id-token: write
172+
contents: read
173+
attestations: read
174+
needs: [account-terraform-plan, account-terraform-approval]
175+
if: ${{ !cancelled() && needs.account-terraform-plan.result == 'success' && needs.account-terraform-plan.outputs.account_infra_changed == 'true' && needs.account-terraform-approval.result == 'success' }}
176+
runs-on: ubuntu-latest
177+
timeout-minutes: 30
178+
environment:
179+
name: ${{ inputs.environment }}
180+
env:
181+
ACCOUNT_TERRAFORM_ENVIRONMENT: ${{ inputs.environment }}
182+
steps:
183+
- name: Checkout
184+
uses: actions/checkout@0c366fd6a839edf440554fa01a7085ccba70ac98
185+
with:
186+
ref: ${{ needs.account-terraform-plan.outputs.plan_sha }}
187+
188+
- name: Connect to AWS
189+
uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
190+
with:
191+
aws-region: eu-west-2
192+
role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/auto-ops
193+
role-session-name: ${{ format('github-actions-{0}-{1}-{2}', github.run_id, github.run_attempt, github.job) }}
194+
195+
- uses: hashicorp/setup-terraform@5e8dbf3c6d9deaf4193ca7a8fb23f2ac83bb6c85
196+
with:
197+
terraform_version: ${{ env.ACCOUNT_TERRAFORM_VERSION }}
198+
199+
- name: Resolve account terraform state bucket
200+
id: account-state-bucket
201+
run: echo "bucket_name=$(bash ./utilities/scripts/resolve_account_terraform_state_bucket.sh)" >> "$GITHUB_OUTPUT"
202+
203+
- name: Retrieve Account Terraform Plan
204+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
205+
with:
206+
name: ${{ env.ACCOUNT_TERRAFORM_ARTIFACT_NAME }}
207+
path: infrastructure/account
208+
209+
- name: Verify Account Terraform Plan Attestation
210+
env:
211+
GH_TOKEN: ${{ github.token }}
212+
run: |
213+
gh attestation verify infrastructure/account/tfplan \
214+
--repo "$GITHUB_REPOSITORY" \
215+
--signer-workflow "$GITHUB_REPOSITORY/.github/workflows/account-terraform.yml"
216+
217+
- name: Terraform Init (account)
218+
working-directory: infrastructure/account
219+
env:
220+
ACCOUNT_TERRAFORM_BUCKET_NAME: ${{ steps.account-state-bucket.outputs.bucket_name }}
221+
run: make init ENVIRONMENT="$ACCOUNT_TERRAFORM_ENVIRONMENT" BUCKET_NAME="$ACCOUNT_TERRAFORM_BUCKET_NAME"
222+
223+
- name: Terraform Apply (account)
224+
# Ignore cancellations to prevent Terraform from being killed while it holds a state lock
225+
# A stuck process can still be killed with the force-cancel API operation
226+
if: ${{ !failure() }}
227+
working-directory: infrastructure/account
228+
run: make apply-ci ENVIRONMENT="$ACCOUNT_TERRAFORM_ENVIRONMENT"

‎.github/workflows/pr-deploy-and-test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ jobs:
2121
with:
2222
apigee_environment: internal-dev
2323
build_recordprocessor_image: ${{ github.event.action == 'opened' || github.event.action == 'reopened' }}
24-
diff_base_sha: ${{ github.event.before }}
24+
diff_base_sha: ${{ github.event.action == 'synchronize' && github.event.before || github.event.pull_request.base.sha }}
2525
diff_head_sha: ${{ github.event.pull_request.head.sha }}
2626
run_diff_check: ${{ github.event.action == 'synchronize' }}
2727
create_mns_subscription: true

‎infrastructure/account/Makefile‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,13 @@ tf_cmd = AWS_PROFILE=$(AWS_PROFILE) terraform
66
tf_state= -backend-config="bucket=$(BUCKET_NAME)"
77
tf_vars= -var-file=environments/$(ENVIRONMENT)/variables.tfvars
88

9+
define require_bucket_name
10+
@if [ -z "$(strip $(BUCKET_NAME))" ]; then \
11+
echo "BUCKET_NAME variable not set. Use 'make init ENVIRONMENT=... BUCKET_NAME=...'"; \
12+
exit 1; \
13+
fi
14+
endef
15+
916
.PHONY: lock-provider workspace init plan apply clean destroy output tf-%
1017

1118
lock-provider:
@@ -17,17 +24,25 @@ workspace:
1724
$(tf_cmd) workspace select -or-create $(ENVIRONMENT) && echo "Switched to workspace/environment: $(ENVIRONMENT)"
1825

1926
init:
27+
$(require_bucket_name)
2028
$(tf_cmd) init $(tf_state) -upgrade $(tf_vars)
2129

2230
init-reconfigure:
31+
$(require_bucket_name)
2332
$(tf_cmd) init $(tf_state) -upgrade $(tf_vars) -reconfigure
2433

2534
plan: workspace
2635
$(tf_cmd) plan $(tf_vars)
2736

37+
plan-ci: workspace
38+
$(tf_cmd) plan $(tf_vars) -out=tfplan -input=false
39+
2840
apply: workspace
2941
$(tf_cmd) apply $(tf_vars) -auto-approve
3042

43+
apply-ci: workspace
44+
$(tf_cmd) apply -input=false tfplan
45+
3146
clean:
3247
rm -rf build .terraform upload-key
3348

‎infrastructure/account/auto_ops_policy.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,7 @@
214214
"iam:DeleteGroupPolicy",
215215
"iam:ListMFADeviceTags",
216216
"elasticache:*",
217+
"shield:*",
217218
"iam:DeletePolicyVersion",
218219
"chatbot:*"
219220
],

‎infrastructure/account/recordprocessor_ecr_repo.tf‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ resource "aws_ecr_lifecycle_policy" "recordprocessor_repository_lifecycle_policy
1414
"rules": [
1515
{
1616
"rulePriority": 1,
17-
"description": "Keep last 10 images",
17+
"description": "Keep last 10 images.",
1818
"selection": {
1919
"tagStatus": "any",
2020
"countType": "imageCountMoreThan",

‎utilities/scripts/resolve_account_terraform_state_bucket.sh‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/bin/bash
2+
3+
set -euo pipefail
4+
5+
read -r configured_bucket <<< "${CONFIGURED_ACCOUNT_TERRAFORM_STATE_BUCKET:-}"
6+
read -r state_bucket_environment <<< "${ACCOUNT_TERRAFORM_STATE_ENVIRONMENT:-}"
7+
8+
[ -n "$configured_bucket" ] && printf '%s\n' "$configured_bucket" && exit 0
9+
10+
[ -n "$state_bucket_environment" ] || {
11+
echo "ACCOUNT_TERRAFORM_STATE_ENVIRONMENT must be set when ACCOUNT_TERRAFORM_STATE_BUCKET is not configured." >&2
12+
exit 1
13+
}
14+
15+
case "$state_bucket_environment" in
16+
internal-dev|internal-qa|preprod|prod)
17+
;;
18+
*)
19+
echo "ACCOUNT_TERRAFORM_STATE_ENVIRONMENT must be one of: internal-dev, internal-qa, preprod, prod." >&2
20+
exit 1
21+
;;
22+
esac
23+
24+
printf 'immunisation-%s-terraform-state-files\n' "$state_bucket_environment"

0 commit comments

Comments
 (0)