VED-1103: Refactor Batch Processor (recordprocessor) deployment to a dedicated pipeline stage (#1279)

* Implement deployment workflow for Recordprocessor and update backend deployment dependencies

- Added a new GitHub Actions workflow for deploying the Recordprocessor batch processor.
- Updated the continuous deployment workflow to include a dependency on the new Recordprocessor deployment.
- Introduced a new variable for the Recordprocessor image tag in Terraform configuration.
- Refactored ECS task definition to use the new image tag variable for the Recordprocessor container.

* Update AWS credentials configuration in deployment workflows

- Explicitly set the AWS_INFRA_ROLE_ARN secret in the continuous deployment workflow.
- Upgrade aws-actions/configure-aws-credentials and aws-actions/amazon-ecr-login to specific versions for improved stability.

* Refactor deployment workflows to streamline Recordprocessor integration

- Removed the separate Recordprocessor deployment workflow and integrated its functionality into the backend deployment workflow.
- Added a new input parameter `enable_recordprocessor` to conditionally build and push the Recordprocessor Docker image.
- Updated dependencies in the backend deployment to ensure proper execution order and image tagging for the Recordprocessor.
- Enhanced the continuous deployment workflow by simplifying dependencies and improving clarity.

* Update ECR repository check in deployment workflow to fail on missing repository

- Changed the behavior of the ECR repository check to fail the build-and-push-recordprocessor job if the repository does not exist, instead of skipping the image build and push. This ensures that the deployment process is more robust and provides clearer feedback on repository availability.

* Update image tag resolution in backend deployment workflow

- Modified the backend deployment workflow to include a new step for resolving the image tag based on the `enable_recordprocessor` input. This change ensures that the image tag is set to the output of the build-and-push step if the Recordprocessor is enabled, otherwise defaults to "latest". This enhances the clarity and reliability of the image tagging process during deployment.

* change rollback

* Rollback enable_recordprocessor variable and create ECR repo if it doesn't exist

* Add permissions to build-and-push job

* Update AWS connection

* Add environment inputs

* Refactor Dockerfile paths and update deployment workflow

- Adjusted the Dockerfile to use correct paths for Poetry files and source code within the `lambdas/recordprocessor` directory.
- Updated the deployment workflow to specify the Dockerfile location during the build process, ensuring proper image creation for the Recordprocessor.

* Fix Dockerfile paths and streamline deployment workflow

- Updated paths in the Dockerfile to correctly reference the `recordprocessor` directory for Poetry files and source code.
- Adjusted the deployment workflow to ensure the Docker build command points to the correct location of the Dockerfile, enhancing the image build process.

* Update Docker build context in deployment workflow

- Changed the working directory for the Docker build command to the root of the repository, ensuring that all necessary files are included during the image build process.
- Adjusted the command to create the ECR repository to ensure it is correctly formatted, enhancing the reliability of the deployment workflow.

* Refactor Docker image build process in deployment workflow

- Removed the `enable_recordprocessor` input from the continuous deployment workflow.
- Added a conditional check in the backend deployment workflow to skip the Docker build and push if the image already exists in ECR, improving efficiency and reducing unnecessary builds.

* Enhance deployment workflow with Terraform planning step

- Introduced a new `terraform-plan` job in the deployment workflow to manage infrastructure changes before applying them.
- Added steps for AWS connection, Terraform initialization, and planning, ensuring a structured approach to infrastructure management.
- Updated the `build-and-push-recordprocessor` job to depend on the `terraform-apply` job, streamlining the deployment process.
- Removed redundant image tag output handling, simplifying the workflow logic.

* Add ECR repository import step to deployment workflow

- Introduced a new step to import the ECR repository into Terraform state if it exists, enhancing the deployment process.
- This addition ensures that the workflow can manage existing resources more effectively, preventing potential conflicts during deployment.

* Remove ECR repository import step from deployment workflow

- Eliminated the step that imported the ECR repository into Terraform state, streamlining the deployment process.
- This change simplifies the workflow by removing unnecessary checks for existing resources, focusing on a more straightforward deployment approach.

* Refactor deployment workflow and ECR repository configuration

- Updated the deployment workflow to use a new naming convention for the Docker image tag and repository.
- Removed the conditional check for existing images in ECR, ensuring a build and push occurs every time.
- Introduced a new ECR repository resource for the record processor, enhancing image management.
- Adjusted IAM policies and ECS task definitions to reference the new repository configuration.

* Update infrastructure/instance/ecs_batch_processor_config.tf

Co-authored-by: Ed Hall <239591530+edhall-nhs@users.noreply.github.com>

* Update infrastructure/account/recordprocessor_ecr_repo.tf

Co-authored-by: Ed Hall <239591530+edhall-nhs@users.noreply.github.com>

* Refactor deployment workflow to reorder jobs and enhance Terraform planning

- Moved the `terraform-plan` job to depend on the `build-and-push-recordprocessor` job, ensuring a more logical sequence in the deployment process.
- Retained the steps for AWS connection, Terraform initialization, and planning within the `terraform-plan` job, maintaining infrastructure management integrity.
- Removed the previous `terraform-plan` job definition to streamline the workflow.

* Update ECS batch processor configuration to use project short name for ECR repository

* Add support for conditional recordprocessor image builds in deployment workflows

- Introduced a new input parameter `build_recordprocessor_image` in the deployment workflows to control the building of the recordprocessor Docker image.
- Updated the `deploy-backend.yml` to include logic for building the image based on changes detected in the PR.
- Added a new job in `pr-deploy-and-test.yml` to detect changes in the recordprocessor directory, influencing the deployment process.
- Enhanced the handling of image tags to default to 'latest' if no build is triggered, improving deployment flexibility.

* Enhance deployment workflows with recordprocessor change detection

- Added a new job to detect changes in the recordprocessor directory, influencing the build process for the Docker image.
- Updated the deployment workflows to conditionally build the recordprocessor image based on detected changes.
- Improved tagging of Docker images to include a 'latest' tag alongside the versioned tag for better image management.
- Adjusted dependencies in the deployment jobs to ensure proper sequencing and execution based on change detection.

* Update deployment workflows to enhance recordprocessor image tagging

- Modified the logic for setting the `TF_VAR_recordprocessor_image_tag` environment variable to include the `sub_environment` in the image tag format.
- This change improves the clarity and traceability of Docker image versions during deployment.

* temp commit to rebuild image

* Enhance deployment workflows for recordprocessor image management

- Added output for `recordprocessor_image_tag` in `deploy-backend.yml` to streamline image tagging.
- Updated logic in `pr-deploy-and-test.yml` to conditionally build the recordprocessor image based on detected changes, improving deployment efficiency.
- Refined steps for resolving existing image tags, ensuring proper handling when no new image is built.

* chore: ado build kickstart

EmptyCommit:

* Enhance deployment workflow for recordprocessor image management

- Added a new job to resolve existing recordprocessor image tags when not building a new image, improving flexibility in deployment.
- Updated job dependencies to ensure proper sequencing between building and resolving image tags.
- Refined output handling for `recordprocessor_image_tag` to accommodate both build and resolve scenarios, enhancing clarity in image management.

* chore: ado build kickstart

EmptyCommit:

* - Added a conditional check to the deployment job to ensure it only runs if the previous terraform plan was successful and not cancelled, improving workflow reliability.

* Refactor GitHub Actions workflows for improved change detection and quality checks

- Updated change detection logic in `continuous-deployment.yml` and `pr-deploy-and-test.yml` to include `shared/src/common/` directory.
- Added a new job for quality checks in `pr-deploy-and-test.yml` to enhance code quality before deployment.
- Cleaned up whitespace in `deploy-backend.yml` for better readability.
- Minor update to error logging in `batch_processor.py` for clarity.

* Update ECR repository configuration for recordprocessor

- Renamed the ECR repository from `processing_repository` to `recordprocessor_repository` for clarity.
- Added a lifecycle policy to manage images, retaining the last 10 images to optimize storage and maintain relevant versions.

* chore: ado build kickstart

EmptyCommit:

* Fix change detection logic in continuous deployment workflow to include the correct path for shared resources

* Update change detection logic in GitHub Actions workflow to include the correct path for shared resources

* Refine change detection logic in continuous deployment workflow to exclude shared resources

* Improve change detection logic in continuous deployment workflow to ensure accurate path matching for recordprocessor changes

* deploy step testing

* deploy testing

* deploy test

* deploy test

* deployment test

* build testing

* deploy test

* Fix logging message in file level validation to ensure clarity on processing cessation.

* Enhance CI workflows by refining change detection logic and re-enabling quality checks for PR deployments

* Move detect changes job to deploy-backend

* Revert "Move detect changes job to deploy-backend"

This reverts commit 8cbf3ae.

* Refactor CI workflows to centralize recordprocessor change detection and streamline deployment conditions

* Update deploy-backend workflow to use environment variables for recordprocessor change detection

* Update deploy-backend workflow to make recordprocessor image build required

* VED-1116: Update PR Teardown to delete PR recordprocessor images (#1303)

* Add cleanup step for recordprocessor ECR images in PR teardown workflow

* Refactor PR teardown workflow to queue image deletions for recordprocessor ECR

* VED-1120: Add option to specify which recordprocessor image to use on manual runs (#1301)

* Enhance deploy-backend workflow to allow optional recordprocessor image tag input. Update logic to handle provided image tag during deployment, ensuring proper validation when building the image is disabled.

* Clarify description for recordprocessor image tag in deploy-backend workflow, specifying its requirement when image building is disabled.

* Remove conditional check for workflow dispatch in AWS connection step of deploy-backend workflow.

* Enhance backend deployment workflow and infrastructure configuration

- Added `recordprocessor_image_tag` input to the backend deployment workflow for better image management.
- Implemented checks for the existence of base and head SHA references to ensure valid deployments.
- Updated ECR lifecycle policy to retain the last 100 images instead of 10.
- Modified `recordprocessor_image_tag` variable to require a non-empty value, improving validation for CI/CD processes.

* Refactor deployment workflows and update variable names

- Renamed `recordprocessor_diff_base_sha`, `recordprocessor_diff_head_sha`, and `run_recordprocessor_diff_check` to `diff_base_sha`, `diff_head_sha`, and `run_diff_check` for consistency across workflows.
- Introduced a new workflow for deploying Lambda artifacts, enhancing deployment flexibility.
- Updated ECR lifecycle policy to retain the last 50 images instead of 100.
- Changed the `recordprocessor_image_tag` variable to `recordprocessor_image_uri` for clarity and improved validation.

* Enhance Lambda deployment workflow by adding environment variables

- Introduced `ENVIRONMENT` and `SUB_ENVIRONMENT` variables to the Lambda deployment workflow for improved configuration management.
- Refactored manifest file generation to utilize `jq` for better JSON handling and readability, replacing the previous heredoc method.
- Updated output handling for Terraform variable names to ensure proper formatting.

* Remove inherited secrets from backend deployment workflow for improved security and clarity.

* Refactor deployment workflows by removing unnecessary variables and consolidating job outputs

- Removed `terraform_variable_name` and `manifest_name` from the backend deployment workflow for clarity.
- Updated the Lambda deployment workflow to streamline output handling and improve variable management.
- Consolidated job outputs to enhance readability and maintainability of the workflows.

* Update Docker context path in backend deployment workflow for improved structure

* Refactor Lambda deployment workflow to improve path handling

- Replaced string concatenation for `diff_paths` with an array to manage paths more effectively.
- Updated condition checks for deployment mode to utilize the new array structure, enhancing clarity and maintainability.

* Remove redundant AWS ECR image description check from Lambda deployment workflow to streamline the process and improve efficiency.

* Enhance Lambda deployment workflow to check for existing image digest before building and pushing Docker images. This prevents unnecessary builds and optimizes the deployment process by reusing existing images when available.

* Update ECR lifecycle policy to retain the last 10 images instead of 50, optimizing image management and storage.

* Enhance Lambda deployment workflow by adding support for implicit tag prefix reuse. Introduced a new input parameter to allow resolving the latest tag when the image version is not provided, improving flexibility in deployment configurations. Additionally, updated the manifest generation to include more contextual information such as build timestamp and Git metadata.

* Update continuous deployment and backend deployment workflows to include new parameters for image processing. Added support for building the record processor image and specifying the image version based on the Git SHA, enhancing deployment flexibility and traceability.

* Update deployment workflows to include output for record processor image version. Adjusted dependencies in the continuous deployment workflow to ensure proper image versioning based on previous job outputs, enhancing deployment accuracy.

* Update deployment workflows to disable record processor image build in continuous deployment. Adjusted output variable for image version in backend deployment workflow and refined implicit tag prefix reuse logic. Added AWS connection step in Lambda artifact deployment workflow, enhancing deployment efficiency and flexibility.

* Enhance PR teardown workflow to include fallback for record processor image URI. Added logic to set a placeholder value if the image output is unavailable, ensuring variable validation during destruction processes remains functional.

* Update deployment workflows to disable diff check in continuous deployment and introduce conditional deployment mode for Lambda artifacts. This change enhances flexibility by allowing image reuse when a version is specified, streamlining the deployment process.

* chore: ado test kickstart

EmptyCommit:

---------

Co-authored-by: Ed Hall <239591530+edhall-nhs@users.noreply.github.com>

Author: Thomas-Boyle

.github/workflows/continuous-deployment.yml

@@ -20,6 +20,10 @@ jobs:
     uses: ./.github/workflows/deploy-backend.yml
     with:
       apigee_environment: internal-dev
+      build_recordprocessor_image: false
+      diff_base_sha: ${{ github.event.before }}
+      diff_head_sha: ${{ github.sha }}
+      run_diff_check: true
       create_mns_subscription: true
       environment: dev
       sub_environment: internal-dev
@@ -75,13 +79,17 @@ jobs:
       STATUS_API_KEY: ${{ secrets.STATUS_API_KEY }}
 
   deploy-higher-dev-envs:
-    needs: [run-internal-dev-tests]
+    needs: [run-internal-dev-tests, deploy-internal-dev-backend]
     strategy:
       matrix:
         sub_environment_name: [ref, internal-qa]
     uses: ./.github/workflows/deploy-backend.yml
     with:
       apigee_environment: ${{ matrix.sub_environment_name }}
+      recordprocessor_image_version: ${{ needs.deploy-internal-dev-backend.outputs.recordprocessor_image_version }}
+      diff_base_sha: ${{ github.event.before }}
+      diff_head_sha: ${{ github.sha }}
+      run_diff_check: false
       create_mns_subscription: true
       environment: dev
       sub_environment: ${{ matrix.sub_environment_name }}

.github/workflows/deploy-backend.yml

@@ -6,6 +6,26 @@ on:
       apigee_environment:
         required: true
         type: string
+      build_recordprocessor_image:
+        required: false
+        type: boolean
+        default: false
+      recordprocessor_image_version:
+        required: false
+        type: string
+        default: ""
+      diff_base_sha:
+        required: false
+        type: string
+        default: ""
+      diff_head_sha:
+        required: false
+        type: string
+        default: ""
+      run_diff_check:
+        required: false
+        type: boolean
+        default: false
       create_mns_subscription:
         required: false
         type: boolean
@@ -16,6 +36,10 @@ on:
       sub_environment:
         required: true
         type: string
+    outputs:
+      recordprocessor_image_version:
+        description: Selected immutable image selector used for recordprocessor deployment
+        value: ${{ jobs.deploy-recordprocessor-image.outputs.image_uri }}
   workflow_dispatch:
     inputs:
       apigee_environment:
@@ -39,6 +63,31 @@ on:
           - dev
           - preprod
           - prod
+      build_recordprocessor_image:
+        description: Build and publish a new recordprocessor image
+        required: true
+        type: boolean
+        default: true
+      recordprocessor_image_version:
+        description: Existing immutable recordprocessor image selector (tag, digest, or image URI) for reuse mode
+        required: false
+        type: string
+        default: ""
+      diff_base_sha:
+        description: Base commit SHA for diff checks
+        required: false
+        type: string
+        default: ""
+      diff_head_sha:
+        description: Head commit SHA for diff checks
+        required: false
+        type: string
+        default: ""
+      run_diff_check:
+        description: Enable diff checks to auto-build on code changes
+        required: true
+        type: boolean
+        default: false
       sub_environment:
         type: string
         description: Set the sub environment name e.g. pr-xxx, or green/blue in higher environments
@@ -51,11 +100,40 @@ env: # Sonarcloud - do not allow direct usage of untrusted data
 run-name: Deploy Backend - ${{ inputs.environment }} ${{ inputs.sub_environment }}
 
 jobs:
+  deploy-recordprocessor-image:
+    name: Deploy recordprocessor image
+    uses: ./.github/workflows/deploy-lambda-artifact.yml
+    with:
+      lambda_name: recordprocessor
+      environment: ${{ inputs.environment }}
+      sub_environment: ${{ inputs.sub_environment }}
+      build_image: ${{ inputs.build_recordprocessor_image }}
+      image_version: ${{ inputs.recordprocessor_image_version }}
+      run_diff_check: ${{ inputs.run_diff_check }}
+      diff_base_sha: ${{ inputs.diff_base_sha }}
+      diff_head_sha: ${{ inputs.diff_head_sha }}
+      lambda_paths: |
+        lambdas/recordprocessor/
+      shared_paths: |
+        lambdas/shared/src/common/
+      docker_context_path: lambdas
+      dockerfile_path: lambdas/recordprocessor/Dockerfile
+      ecr_repository: imms-recordprocessor-repo
+      image_tag_prefix: ${{ inputs.sub_environment }}-
+      allow_implicit_tag_prefix_reuse: ${{ inputs.sub_environment == 'internal-dev' || startsWith(inputs.sub_environment, 'pr-') }}
+
   terraform-plan:
     permissions:
       id-token: write
       contents: read
+    needs:
+      - deploy-recordprocessor-image
+    if: ${{ !cancelled() && needs.deploy-recordprocessor-image.result == 'success' }}
+    outputs:
+      recordprocessor_image_uri: ${{ needs.deploy-recordprocessor-image.outputs.image_uri }}
     runs-on: ubuntu-latest
+    env:
+      TF_VAR_recordprocessor_image_uri: ${{ needs.deploy-recordprocessor-image.outputs.image_uri }}
     environment:
       name: ${{ inputs.environment }}
     steps:
@@ -95,7 +173,10 @@ jobs:
       id-token: write
       contents: read
     needs: terraform-plan
+    if: ${{ !cancelled() && needs.terraform-plan.result == 'success' }}
     runs-on: ubuntu-latest
+    env:
+      TF_VAR_recordprocessor_image_uri: ${{ needs.terraform-plan.outputs.recordprocessor_image_uri }}
     environment:
       name: ${{ inputs.environment }}
     steps: