cloud-services.md

Cloud services

Context

• These notes are part of a broader set of principles
• Practices in this section contribute to service reliability
• See also observability
• This is related to:
  • ARCHITECTURE-CLOUD
  • ARCHITECTURE-SUSTAINABILITY
  • SERVICE-TOOLS

Details

• Configure all infrastructure using declarative code such as Terraform and CloudFormation (see everything as code).

• Automate monitoring and alerting (see automate everything and observability.

• Prefer serverless platform as a service (PaaS) over infrastructure as a service (IaaS) (see outsource bottom up).

• Where not serverless use ephemeral and immutable infrastructure.

• Engage your cloud supplier early on in the development process. They have various tools and processes to help you (e.g. AWS Well-Architected Review).

• Understand cloud supplier SLAs.

• Make systems self-healing.

  • Prefer technologies which are resilient by default.
  • Favour global-scoped (e.g. CloudFront or Front Door) or region-scoped services (e.g. S3, Lambda, Azure Functions) to availability-zone (AZ) scoped (e.g. VMs, RDS DBs) or single-instance services (e.g. EC2 instance storage).
  • For AZ-scoped services, use redundancy to create required resilience (e.g. AWS Auto Scaling Groups or Azure Scale/Availability Sets), and:
    • For stateless components use active-active configurations across AZs (e.g. running stateless containers across multiple AZs using AWS Elastic Kubernetes Service)
    • For stateful components, e.g. databases, consider use of active-active configurations across AZs (e.g. Aurora Multi-Master), but be aware of the added complexity conflict resolution for asynchronous replication can bring and potential performance impact where synchronous replication is chosen.
  • Consider use of multiple regions (e.g. for AWS eu-west-1 [Dublin] as well as eu-west-2 [London]) as a way to improve availability, though ensure data sovereignty implications are understood and accepted (see below).
  • Understand failover (e.g. RDS failover) and failed instance replacement times and engineer to accommodate these.

• Be aware of data sovereignty implications of using any systems hosted outside the UK.

  • Make sure your information governance lead is aware and included in decision making.
  • Consider SaaS tools the team uses as well as the systems we build.

• Services should scale automatically up and down.

  • If possible, drive scaling based on metrics which matter to users (e.g. response time), but balance this with the benefits of choosing leading indicators (e.g. CPU usage) to avoid slow scaling from impacting user experience.
  • Understand how rapidly demand can spike and ensure scaling can meet these requirements. Balance scaling needs with the desire to avoid over provisioning and use pre-warming of judiciously where required. Discuss this with the cloud provider well before go live they can assist with pre-warming processes (AWS).

• Infrastructure should always be fully utilised (if it isn't, it's generating waste).

  • Though balance this with potential need to run with some overhead to accommodate failed instance replacement times without overloading remaining instances.

• Keep up to date.

  • Services/components need prompt updates to dependencies where security vulnerabilities are found — even if they are not under active development.
  • Services which use deprecated or unsupported technologies should be migrated onto alternatives as a priority.

• Understand and be able to justify vendor lock in (see outsource from the bottom up).

• Build in governance as a side effect, e.g.

  • Segregate production and non-production workloads.

  • Infrastructure must be tagged to identity the service so that unnecessary resources don't go unnoticed (click to expand)

    AWS Config rule to identify EC2 assets not tagged with "CostCenter" and "Owner":

    {
      "ConfigRuleName": "RequiredTagsForEC2Instances",
      "Description": "Checks whether the CostCenter and Owner tags are applied to EC2 instances.",
      "Scope": {
        "ComplianceResourceTypes": [
          "AWS::EC2::Instance"
        ]
      },
      "Source": {
        "Owner": "AWS",
        "SourceIdentifier": "REQUIRED_TAGS"
      },
      "InputParameters": "{\"tag1Key\":\"CostCenter\",\"tag2Key\":\"Owner\"}"
    }

    Further reading: AWS Config

    TO DO: Azure equivalent

  • Configure audit tools such as CloudTrail.