diff --git a/README.md b/README.md index 2567adb..0bad7f3 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,49 @@ # Temporal Serverless Workers Skill -Deploy and operate [Temporal](https://temporal.io/) Workers on serverless compute with help from a coding agent. The skill guides an agent through the complete AWS Lambda lifecycle: scoping, access checks, Worker implementation, packaging, deployment, Temporal registration, verification, troubleshooting, updates, and rollback. +Deploy and operate [Temporal](https://temporal.io/) Workers on serverless compute with help from a coding agent. The skill guides an agent through the complete lifecycle on AWS Lambda and GCP Cloud Run: scoping, access checks, Worker implementation, packaging, deployment, Temporal registration, verification, troubleshooting, updates, and rollback. > [!WARNING] > This skill is in Public Preview and will continue to evolve. Pin the Temporal SDK, serverless Worker package, and CLI versions for long-lived projects. > [!NOTE] -> Temporal Serverless Workers on AWS Lambda are in Public Preview and are available to all Temporal Cloud customers without an access request. AWS Lambda is currently the only compute provider supported by this skill. +> **AWS Lambda** is in Public Preview and available to all Temporal Cloud customers without an access request. +> **GCP Cloud Run** is in Public Preview and available to all Temporal Cloud customers without an access request. + +> [!IMPORTANT] +> The two providers have different execution models. Lambda invokes a function per unit of work and the Worker exits when the invocation ends. Cloud Run resizes a pool of long-lived instances, scaling to zero when idle. That changes what the Worker code is, what bounds an Activity, what there is to tune, and how failures present — guidance does not transfer between them. ## What the skill can do -- Build Serverless Workers with the Go, Python, TypeScript, Java, or .NET SDK. -- Package and deploy Workers to AWS Lambda with the correct architecture, timeout, and shutdown settings. -- Configure the separate AWS roles used by the Lambda function and by Temporal. +- Build Serverless Workers with the Go, Python, TypeScript, Java, or .NET SDK on either provider, plus Ruby and Rust on Cloud Run. +- Package and deploy to AWS Lambda with the correct architecture, timeout, and shutdown settings — or containerize and deploy to a Cloud Run Worker Pool. +- Configure the two distinct identities each provider needs: execution and invocation roles on AWS, runner and invoker service accounts on GCP. - Register a Worker Deployment Version, validate its Task Queue binding, and set it current. -- Verify a deployment from both Temporal Workflow history and Lambda logs. -- Diagnose Workers that are not invoked or do not complete Tasks. -- Publish immutable Lambda versions, update deployments, and roll back safely. -- Add OpenTelemetry observability with the AWS Distro for OpenTelemetry. +- Verify a deployment from both Temporal Workflow history and the provider's logs. +- Diagnose Workers that are never started, or that start but do not complete Tasks. +- Keep each build immutable, update deployments, and roll back safely. +- Add OpenTelemetry observability — the AWS Distro for OpenTelemetry on Lambda, the SDK's standard setup plus Cloud Logging on Cloud Run. - Configure self-hosted Temporal deployments that meet the serverless prerequisites. ## Support | Area | Supported | |---|---| -| Compute | AWS Lambda — Public Preview | +| Compute | AWS Lambda and GCP Cloud Run — Public Preview | | Temporal | Temporal Cloud and self-hosted Temporal Service | -| SDKs | Go, Python, TypeScript, Java, .NET | +| SDKs | Lambda: Go, Python, TypeScript, Java, .NET. Cloud Run: those plus Ruby and Rust | | Other compute providers | Not currently supported | -For Temporal Cloud, the Namespace must be hosted on AWS. The Namespace and Lambda function may be in different AWS regions. +For Temporal Cloud, the Namespace must be hosted on the same cloud provider as the compute — AWS for Lambda, GCP for Cloud Run. Regions need not match. ## Before you start Before starting, make sure you can sign in to: -- An AWS account with permission to inspect and create the required Lambda, IAM, CloudFormation, and logging resources. -- A Temporal Cloud Namespace hosted on AWS, or a compatible self-hosted Temporal Service. +- For AWS Lambda: an AWS account with permission to inspect and create the required Lambda, IAM, CloudFormation, and logging resources. +- For GCP Cloud Run: a GCP project with the Cloud Run and Artifact Registry APIs enabled, and permission to create Worker Pools, service accounts, and Secret Manager secrets. +- A Temporal Cloud Namespace hosted on the same cloud provider as your compute, or a compatible self-hosted Temporal Service. -You do not need to install or configure the AWS CLI, `tcld`, or the Temporal CLI before you begin. The skill checks what is already available and can help set up the tools and supported login flows needed for the task. If you prefer not to install a CLI, or a login method is unavailable, it can guide you through the corresponding Temporal Cloud UI or AWS console steps instead. It never asks you to paste credentials or secrets into the conversation. +You do not need to install or configure the AWS CLI, `gcloud`, Terraform, `tcld`, or the Temporal CLI before you begin. The skill checks what is already available and can help set up the tools and supported login flows needed for the task. If you prefer not to install a CLI, or a login method is unavailable, it can guide you through the corresponding Temporal Cloud UI, AWS console, or Google Cloud console steps instead. It never asks you to paste credentials or secrets into the conversation. ## Installation @@ -98,23 +103,31 @@ Package this Java Worker as a shaded jar and deploy it to Lambda. Deploy this .NET Worker to Lambda with a runtime-specific publish. ``` +```text +Deploy this Go Worker to a GCP Cloud Run Worker Pool. +``` + +```text +My Cloud Run Worker Pool is stuck at zero instances. Find out why. +``` + For a new deployment, the skill follows five stages: 1. **Scope** — confirm the SDK, compute provider, Namespace, region, and resource-naming prefix. -2. **Access** — verify AWS and Temporal identities and permissions, then present the exact billable resources for approval. -3. **Build** — install the serverless Worker package, inspect its current API, author the Worker, and deploy it. -4. **Connect** — configure Temporal's invocation role, register the Worker Deployment Version, validate the Task Queue binding, and set the version current. +2. **Access** — verify cloud-provider and Temporal identities and permissions, then present the exact billable resources for approval. +3. **Build** — author the Worker and deploy it. On Lambda that means installing the serverless Worker package and inspecting its current API; on Cloud Run it means an ordinary long-lived Worker in a container image. +4. **Connect** — grant Temporal access to the compute (an invocation role on AWS, an impersonated invoker service account on GCP), register the Worker Deployment Version, confirm it is reachable, and set the version current. 5. **Verify and hand back** — run a Workflow, confirm two independent health signals, inventory every created resource, and offer teardown. Nothing is created before you approve the resource list. Troubleshooting and inspection requests skip the deployment walkthrough and begin with read-only diagnostics. ## Important operating constraints -- Serverless Workers and their APIs are Public Preview, not generally available. +- Serverless Workers on AWS Lambda and GCP Cloud Run are Public Preview, not generally available. - Every Workflow must use a Worker Versioning behavior: `Pinned` or `AutoUpgrade`. - The deployment name and build ID in Worker code must exactly match the registered Worker Deployment Version. -- Production releases should map each build ID to one immutable Lambda version. -- Activities must finish within the Lambda invocation limit and configured shutdown buffer; Workflow duration remains unbounded. +- Production releases should map each build ID to one immutable build: a published Lambda version, or a dedicated Cloud Run Worker Pool. +- Activities must finish within the compute provider's execution bounds; Workflow duration remains unbounded. Lambda Activities are bounded by the invocation deadline and shutdown buffer, while Cloud Run Activities can be interrupted during scale-in. See [`references/concepts.md`](references/concepts.md) and the selected provider's SDK and constraints references. - Secrets belong in a secret store for shared or production deployments, not plaintext environment variables. - Temporal creates and manages the Worker Controller Instance (WCI); this skill never creates or manages it directly. @@ -135,6 +148,18 @@ Nothing is created before you approve the resource list. Troubleshooting and ins | [`references/aws-lambda/versioning.md`](references/aws-lambda/versioning.md) | Immutable releases, updates, and rollback | | [`references/aws-lambda/observability.md`](references/aws-lambda/observability.md) | Shared ADOT Collector configuration, X-Ray enablement, and IAM permissions | | [`references/aws-lambda/self-hosted.md`](references/aws-lambda/self-hosted.md) | Self-hosted Temporal prerequisites and configuration | +| [`references/gcp-cloud-run/sdk-go.md`](references/gcp-cloud-run/sdk-go.md) | Go versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/sdk-python.md`](references/gcp-cloud-run/sdk-python.md) | Python versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/sdk-typescript.md`](references/gcp-cloud-run/sdk-typescript.md) | TypeScript versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/sdk-java.md`](references/gcp-cloud-run/sdk-java.md) | Java versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/sdk-dotnet.md`](references/gcp-cloud-run/sdk-dotnet.md) | .NET versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/setup.md`](references/gcp-cloud-run/setup.md) | End-to-end Cloud Run deployment: container image, Worker Pool, registration, verification, teardown | +| [`references/gcp-cloud-run/iam.md`](references/gcp-cloud-run/iam.md) | Operator permissions, runner vs invoker service accounts, and the Terraform module | +| [`references/gcp-cloud-run/constraints.md`](references/gcp-cloud-run/constraints.md) | What follows from Cloud Run's pool-of-instances model — instance lifetime, autoscaling, scale-in interrupting Activities — and what does not generalize | +| [`references/gcp-cloud-run/versioning.md`](references/gcp-cloud-run/versioning.md) | One Worker Pool per build ID, the redeploy-into-a-live-pool hazard, and rollback | +| [`references/gcp-cloud-run/diagnostics.md`](references/gcp-cloud-run/diagnostics.md) | Pool annotations, scaling failures, and Worker-side errors | +| [`references/gcp-cloud-run/observability.md`](references/gcp-cloud-run/observability.md) | Logs and the scaling signals to watch | +| [`references/gcp-cloud-run/self-hosted.md`](references/gcp-cloud-run/self-hosted.md) | Self-hosted prerequisites: dynamic config, the server's GCP identity, invoker creation | | [`assets/`](assets/) | CloudFormation templates for Temporal invocation roles | ## Feedback diff --git a/SKILL.md b/SKILL.md index dcee00d..4753b86 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,34 +1,39 @@ --- name: temporal-serverless -description: 'Deploy and operate Temporal Workers on serverless compute (AWS Lambda) driven by the Worker Controller Instance (WCI). Use when the user mentions: "serverless worker", "Temporal serverless", "Worker Controller Instance", "WCI", "deploy Temporal worker on Lambda", "Lambda packaging", "Lambda timeout", "WCI inspection", "CloudFormation Temporal".' +description: 'Deploy and operate Temporal Workers on serverless compute (AWS Lambda, GCP Cloud Run) driven by the Worker Controller Instance (WCI). Use when the user mentions: "serverless worker", "Temporal serverless", "Worker Controller Instance", "WCI", "deploy Temporal worker on Lambda", "Lambda packaging", "Lambda timeout", "WCI inspection", "CloudFormation Temporal", "Cloud Run worker", "Worker Pool", "deploy Temporal worker on Cloud Run", "gcloud run worker-pools", "invoker service account".' version: 0.6.2 -disable-model-invocation: true --- # Skill: temporal-serverless ## Overview -This skill helps users deploy and operate Temporal Workers on serverless compute. Instead of a long-lived process, Temporal invokes the Worker on demand through the Worker Controller Instance (WCI); the Worker processes available Tasks and shuts down, scaling to zero when idle. The skill produces Worker code, deployment configuration, connection configs, and packaging steps for the chosen SDK, and walks users through troubleshooting when serverless Workers aren't picking up Tasks. +This skill helps users deploy and operate Temporal Workers on serverless compute controlled by the Worker Controller Instance (WCI). On Lambda, the WCI invokes short-lived Workers on demand; on Cloud Run, it resizes a pool of ordinary long-lived Workers. Both models scale to zero when idle. The skill produces Worker code, deployment configuration, connection configs, and packaging steps for the chosen SDK, and walks users through troubleshooting when serverless Workers aren't picking up Tasks. ## Supported compute providers | Cloud provider | Compute service | Support | Reference directory | |---|---|---|---| | AWS | Lambda | Supported — Public Preview, open to all Temporal Cloud customers | `references/aws-lambda/` | -| GCP | Cloud Run | Not supported | — | +| GCP | Cloud Run | Supported — Public Preview, open to all Temporal Cloud customers | `references/gcp-cloud-run/` | Only a provider marked Supported is covered. If a request names another, say it is not supported and stop; do not adapt a supported provider's material to it. **Never let the provider be an unstated assumption:** when the request does not name one, it is confirmed in the step 1 questions, not silently defaulted. -Every supported provider's directory carries the same shared layout — `setup.md`, `iam.md`, `versioning.md`, `diagnostics.md`, `observability.md`, `self-hosted.md` — plus one `sdk-.md` file for each supported SDK. Paths below are written `references//…`; substitute the directory from the table. Provider-specific commands, templates, permissions, SDK APIs, and defaults live there — this file stays at the workflow level. When a step needs concrete commands or SDK details, go to the reference file named at the end of that step. +**The two providers have different execution models.** Lambda invokes a function per unit of work and the Worker exits when the invocation ends. Cloud Run resizes a pool of long-lived instances, scaling to zero when idle. That changes what the Worker code is (a handler against a provider package, versus an ordinary long-lived Worker), what bounds an Activity, what there is to tune, and how failures present. Read `references/concepts.md` and the selected provider's SDK reference before advising on any of it; for Cloud Run, also read `references/gcp-cloud-run/constraints.md`. Never carry a fact from one provider to the other. -| SDK language | AWS Lambda reference | -|---|---| -| Go | `references/aws-lambda/sdk-go.md` | -| Python | `references/aws-lambda/sdk-python.md` | -| TypeScript | `references/aws-lambda/sdk-typescript.md` | -| Java | `references/aws-lambda/sdk-java.md` | -| .NET | `references/aws-lambda/sdk-dotnet.md` | +Every supported provider's directory carries the same shared layout — `setup.md`, `iam.md`, `versioning.md`, `diagnostics.md`, `observability.md`, `self-hosted.md` — plus one `sdk-.md` file for each supported SDK. Cloud Run also has `constraints.md` for its pool lifecycle, autoscaling, and scale-in behavior; Lambda's invocation lifecycle and timeout relationships live in `references/concepts.md` and its SDK references. Paths below are written `references//…`; substitute the directory from the table. Provider-specific commands, templates, permissions, SDK APIs, and defaults live there — this file stays at the workflow level. When a step needs concrete commands or SDK details, go to the reference file named at the end of that step. + +| SDK language | AWS Lambda reference | GCP Cloud Run reference | +|---|---|---| +| Go | `references/aws-lambda/sdk-go.md` | `references/gcp-cloud-run/sdk-go.md` | +| Python | `references/aws-lambda/sdk-python.md` | `references/gcp-cloud-run/sdk-python.md` | +| TypeScript | `references/aws-lambda/sdk-typescript.md` | `references/gcp-cloud-run/sdk-typescript.md` | +| Java | `references/aws-lambda/sdk-java.md` | `references/gcp-cloud-run/sdk-java.md` | +| .NET | `references/aws-lambda/sdk-dotnet.md` | `references/gcp-cloud-run/sdk-dotnet.md` | + +The two columns are not interchangeable. A Lambda SDK reference describes a provider Worker package and its handler; the Cloud Run counterpart describes an ordinary long-lived Worker with Worker Versioning enabled, and there is no Cloud Run Worker package. Read the one for the confirmed provider. + +**Ruby and Rust run on Cloud Run and have no Lambda packages**, so they have no row above. There is no `sdk-ruby.md` or `sdk-rust.md`; for those two, follow `references/gcp-cloud-run/setup.md` and the SDK's own Cloud Run guide, and say that the skill carries no per-SDK reference for them. **Public Preview is not GA.** The APIs are still evolving and may change: pin SDK and CLI versions for anything long-lived, and read the installed package's actual API surface rather than writing from memory. @@ -42,7 +47,7 @@ Follow these steps in order. Each step is provider-neutral; the concrete command > Here's what's about to happen, before I ask anything: > -> - This creates real resources in your cloud account — the compute unit that runs your Worker, roles, an infrastructure stack, logs. They're live and billable for as long as they exist. +> - This creates real resources in your cloud account — the compute unit that runs your Worker, identities and access grants, infrastructure state, and logs. They're live and billable for as long as they exist. > - **How it goes.** Five stages: > - **Scope** — a handful of questions, below. > - **Access** — check credentials and permissions on both sides, then show you an exact list of what I'm about to create and wait for your approval. @@ -54,7 +59,7 @@ Follow these steps in order. Each step is provider-neutral; the concrete command **Write the summary provider-neutral, because at that point you do not know the provider.** It is one of the things step 1 asks. Say "your cloud account", never the name of a provider you have not been told. The same applies to the account, Namespace, and region: if a cheap read-only call has already told you (see step 1), name what you actually found; otherwise leave it out rather than filling it in with a plausible guess. -Skip the summary for troubleshooting, inspection, and configuration-change tasks. Someone whose Worker is not being invoked does not need an overview of a deployment they have already done. +Skip the summary for troubleshooting, inspection, and configuration-change tasks. Someone whose Worker is not receiving Tasks does not need an overview of a deployment they have already done. **Then track the run on a checklist, and reprint it every time a step completes.** The eight steps group into the five stages below. Create one item per step, grouped under its stage, and build the checklist as soon as step 1's answers land, so items can name the confirmed provider and the agreed prefix instead of hedging. @@ -76,7 +81,7 @@ Where the harness has a todo list, use it *in addition to* the printed checklist > > **Connect** > ⬜ Create the role Temporal assumes to invoke the Worker -> ⬜ Register the Worker Deployment Version, confirm the validation invocation bound the Task Queue, set it current +> ⬜ Register the Worker Deployment Version, confirm its registration bootstrap bound the Task Queue, set it current > > **Verify and hand back** > ⬜ Start a Workflow and confirm it executes, from both the Temporal side and the provider's logs @@ -92,9 +97,11 @@ Where the harness has a todo list, use it *in addition to* the printed checklist **A step is complete when its verification passed — not when its command exited zero.** Several commands in this workflow exit clean having done nothing: the traffic-shifting and key-revocation commands no-op when their confirmation prompt goes unanswered, and providers return from create and update calls while the resource is still settling. Check an item off against state you read back, not against an exit code. When a step's verification fails, say which step you are on and what it is blocked on rather than moving down the list. -1. **Scope the task.** Identify the SDK language (Go, Python, TypeScript, Java, or .NET), the deployment target (Temporal Cloud or self-hosted — self-hosted has its own server prerequisites), the compute provider, and whether this is a new setup, a configuration change, or troubleshooting. Confirm the deployment target is compatible with the chosen provider — see "A Namespace on the target cloud provider is required" under Provider-neutral principles. Ensure a Temporal client/CLI is available and authenticated to the target. Each changes the specifics. → `references/concepts.md` for what the user is building; `references//setup.md` for the compatibility and client-setup details. +1. **Scope the task.** Identify the SDK language (Go, Python, TypeScript, Java, .NET, and on Cloud Run also Ruby or Rust — **SDK support differs by provider**), the deployment target (Temporal Cloud or self-hosted — self-hosted has its own server prerequisites), the compute provider, and whether this is a new setup, a configuration change, or troubleshooting. Confirm the deployment target is compatible with the chosen provider — see "A Namespace on the target cloud provider is required" under Provider-neutral principles. Ensure a Temporal client/CLI is available and authenticated to the target. Each changes the specifics. → `references/concepts.md` for what the user is building; `references//setup.md` for the compatibility and client-setup details. + + **Ask the compute provider as a real question now that there are two, and carry each option's support status in its description** — AWS Lambda and GCP Cloud Run are both Public Preview and open to all Temporal Cloud customers. Skip the question only when the request already names a provider. Do not restate any of this in a paragraph before the questions; the option description is where it belongs. - **Put the compute provider in that batch of questions as a confirmable default, not a free choice.** Pre-select the supported provider from the table above and carry its support status in the option's description. The user confirms rather than chooses, so it costs no extra turn, but the provider is never something they were assumed into. Skip the question only when the request already names a provider. Do not restate any of this in a paragraph before the questions; the option description is where it belongs. + **The Namespace usually settles it, so ask them together.** A Serverless Worker runs only on the cloud provider hosting its Namespace, so a user with only AWS Namespaces has no Cloud Run option. Where the user genuinely has both, the deciding factors include **Activity duration** (anything over Lambda's 15-minute ceiling rules Lambda out) and **SDK** (Ruby and Rust are Cloud Run only). Say which factor decided it rather than presenting the choice as arbitrary. **Let the user pick the Namespace from a list; never make them retype one.** Namespace names are long and error-prone — a generated suffix on an account ID, `-.`. Where control-plane access is available, `tcld namespace list` returns the full Namespace objects, so one call gives every name with its region — and a region ID is provider-prefixed (`aws-…`, `gcp-…`), so the same response tells you each Namespace's provider. Only the prefix carries meaning; the region itself imposes no constraint. @@ -139,19 +146,38 @@ Where the harness has a todo list, use it *in addition to* the printed checklist **Do not self-select a row.** Drop to a lower one only after the choice above has been put to the user and the browser path chosen, or the login attempted and failed. When you hand off a runbook, say the offer stands — if the user authenticates and comes back, take the work over rather than leaving them to run the steps by hand. - **Before the first account-mutating command, list what you are about to create — with final names — and get approval.** Name the target account and region, then every resource: compute unit, execution role, infrastructure stack, log group, deployment name, and Task Queue. Say plainly that they are live and billable. This is the mirror of the inventory in step 8, and it is worth more here than there: it makes the naming prefix concrete while changing it is still free, and the deployment name, build ID, and Task Queue become expensive to change once step 3 compiles them into the Worker. Skip it only when nothing will be created — a troubleshooting or inspection task. + **Before the first account-mutating command, list what you are about to create — with final names — and get approval.** Name the target account or project and region, then every resource. For Lambda this includes the function, execution and invocation roles, infrastructure stack, and log group. For Cloud Run this includes the image repository and image, Worker Pool, runner and invoker service accounts, Terraform state, logs, and any secrets. For either provider include the Temporal deployment name, build ID, and Task Queue. Say plainly that the resources are live and billable. This is the mirror of the inventory in step 8, and it is worth more here than there: it makes the naming prefix concrete while changing it is still free, and the deployment name, build ID, and Task Queue become expensive to change once step 3 compiles them into the Worker. Skip it only when nothing will be created — a troubleshooting or inspection task. + +3. **Author the Worker.** Follow the selected provider's Worker model; do not infer one from the phrase "serverless Worker." + + - **AWS Lambda:** install the SDK's serverless Worker package before writing code. It is usually shipped separately from the main SDK, and its Public Preview API and handler shape can drift. Read the installed package's actual API before writing the entry point. + - **GCP Cloud Run:** write an ordinary long-lived Worker that starts polling when the container starts. There is no Cloud Run serverless Worker package or per-invocation handler. For Go, Python, TypeScript, Java, and .NET, use the selected `references/gcp-cloud-run/sdk-.md`; for Ruby or Rust, use `references/gcp-cloud-run/setup.md` and the SDK's current Cloud Run guide. -3. **Author the Worker.** *Install the SDK's serverless Worker package before writing any code* — it is usually shipped separately from the main SDK — sometimes on its own version line, sometimes in lockstep with it, and in one SDK not separately at all — so having the base SDK installed does not mean it is importable. Then read the installed package's actual API surface and write against that; these are Public Preview APIs that drift between versions, and generating code from memory costs a build cycle. Entry-point names are not consistent between SDKs, so inspect first rather than pattern-matching from another language. Every Workflow must declare a versioning behavior (`Pinned` or `AutoUpgrade`), per-Workflow or as a Worker-level default — code without it fails at runtime. → `references//sdk-.md` (package, install, API inspection, entry point, handler shape, versioning behavior, tuned defaults). + Every Workflow must declare a versioning behavior (`Pinned` or `AutoUpgrade`), per Workflow or as a Worker-level default. The deployment name and build ID in the Worker must match the version that will be registered. → `references/concepts.md` and the applicable SDK reference; for Cloud Run, also `references/gcp-cloud-run/constraints.md`. -4. **Package and deploy the compute unit.** Build and package per SDK, deploy the compute unit, and set the invocation deadline high enough for the Worker to start, connect, register the Task Queue, and shut down gracefully. Match the build's target architecture to the deployed compute unit's — a mismatch fails only at invocation time, not at build time. After a create or update, wait for the compute unit to reach a ready state before the next step; providers return from these calls while the unit is still settling. → `references//sdk-.md` (build, packaging, runtime, handler, architecture, and SDK-specific deployment values) and `references//setup.md` (shared deployment lifecycle). +4. **Package and deploy the compute unit.** Build for the target runtime and architecture, deploy an immutable unit for this build ID, and wait for the provider to report it ready before registering it; provider create and update calls may return while the resource is still settling. -5. **Grant Temporal permission to invoke the Worker.** Configure the compute provider's access so Temporal can invoke and inspect the Worker. This access is separate from the compute unit's own execution role — do not confuse the two. Two things to get right before you create anything: (a) this grant is **shared, account-wide infrastructure** that a previous deployment may already have created — look for an existing one and extend it to cover your new Worker rather than creating a parallel copy, and never delete or repurpose one you did not create without asking; (b) scope the grant so that *future* immutable builds are covered, not just today's — a grant pinned to one build breaks the next release in a way that surfaces later as an unrelated-looking invocation failure. → `references//iam.md`. + - **AWS Lambda:** package the SDK-specific handler, configure the function architecture and runtime, and set the invocation deadline and shutdown settings together so the Worker can initialize, process Tasks, and drain safely. + - **GCP Cloud Run:** build and push a container image, then create a dedicated Worker Pool for the build ID at zero instances. The container entry point runs the Worker continuously. There is no invocation deadline or Lambda shutdown buffer to configure. -6. **Register the Worker Deployment Version, verify the validation invocation, then set it current.** Create the Worker Deployment Version with the compute provider configured; the deployment name and build ID must exactly match the values in the Worker code. Creating it triggers one validation invocation — **check that it bound the Task Queue before going further.** If the Task Queue is bound, the permission grant, package, config, and deadline are all provably correct, and any later failure is downstream; if it is not, setting the version current will not fix it. Then set it current: through the UI this happens automatically, through the CLI it is a separate step, without which Tasks never route to the version. → `references//setup.md`. + → `references//sdk-.md`, when one exists, and `references//setup.md`. -7. **Verify.** Start a Workflow on the Task Queue and confirm Temporal invokes the Worker — check the Workflow history in the Temporal UI and the compute provider's logs. If it does not progress, → `references//diagnostics.md`. +5. **Grant Temporal permission to control the compute.** Keep the identity Temporal uses separate from the identity the compute runs as. -8. **Hand back the inventory first; offer teardown as the closing note.** The order is inventory → offer, never the reverse. Close with what now exists — compute unit and published build identifiers, roles, infrastructure stacks, region, deployment name and build ID — and what the run actually did, including anything you worked around or deviated from. Say plainly that it is live and billable. These names are only knowable from the run that created them, and reconstructing them later means scanning the user's account. + - **AWS Lambda:** configure the invocation role Temporal assumes to inspect and invoke the function; it is distinct from the function's execution role. + - **GCP Cloud Run:** configure the invoker service account Temporal impersonates with permission to read and update Worker Pools; it is distinct from the runner service account attached to pool instances. + + Treat this grant as shared account- or project-level infrastructure: look for an existing compatible grant before creating another, cover future immutable builds rather than only today's target, and never delete or repurpose infrastructure you did not create without asking. → `references//iam.md`. + +6. **Register the Worker Deployment Version, verify its registration bootstrap, then set it current.** Create the version with compute configured; the deployment name and build ID must exactly match the Worker. The WCI first validates the provider configuration, then starts enough compute for the Worker to poll and register its Task Queues: Lambda emits an invoke action, while Cloud Run resizes the Worker Pool to the scaler's planned count, with a minimum of one for registration even when `initial_count` is zero. This bootstrap happens during version registration and does not require the version to be current. + + Wait until `describe-version` shows the expected Task Queue types before shifting traffic. A binding proves that a Worker started, connected, and polled under the registered deployment name and build ID. If it is absent, setting the version current does not repair the bootstrap; inspect the provider-specific failure path. Do not confuse this with the UI's **Validate Connection** action: for Cloud Run that action reads the pool but neither starts an instance nor tests the update permission used for scaling. + + Then set the version current. The UI does this automatically during its creation flow; the CLI requires a separate command. → `references//setup.md` and `references//diagnostics.md`. + +7. **Verify.** Start a Workflow on the Task Queue and confirm it progresses in Temporal, then confirm the matching provider-side signal: a Lambda invocation and Worker logs, or a Cloud Run pool resize followed by instance startup and Worker logs. If the Workflow does not progress, follow `references//diagnostics.md` rather than translating the other provider's symptoms. + +8. **Hand back the inventory first; offer teardown as the closing note.** The order is inventory → offer, never the reverse. Close with what now exists — compute unit and immutable build identifiers, cloud identities and access grants, infrastructure stack or Terraform state, region, deployment name, and build ID — and what the run actually did, including anything you worked around or deviated from. Say plainly that it is live and billable. These names are only knowable from the run that created them, and reconstructing them later means scanning the user's account or project. **Do not write a teardown script before the user asks for one.** Generating it unprompted buries the inventory under a file they did not request, and the inventory is what they need in order to decide. End with a single line — *"Let me know if you want a teardown script to remove these resources"* — and stop there. Write the script, or run the teardown, when they take you up on it. → `references//setup.md` (Teardown). @@ -169,18 +195,18 @@ How to move through the workflow above. > > **Creating the invocation role Temporal assumes, with a generated External ID** - For anything that creates, updates, or deletes, name the resource and the target account or Namespace explicitly — an approval prompt should arrive with its justification already on screen, not after it. + For anything that creates, updates, or deletes, name the resource and the target account, project, or Namespace explicitly — an approval prompt should arrive with its justification already on screen, not after it. - **Read the current state instead of recalling it.** Check the installed package's API, the CLI's own `--help` for the flags you are about to pass, the compute unit's reported state, and the CLI version. Each of these has drifted in practice: a Public Preview SDK whose fields moved, a CLI too old to have the serverless subcommand at all, a resource that reports success while still settling. - **Do not chain `cd` with commands that create or modify files.** A compound `cd && ` triggers a manual approval prompt no matter how the user's permissions are configured, so scaffolding a project this way asks for approval on every run. Use absolute paths, or the tool's own directory flag (`go -C …`), and rely on the shell's working directory persisting between calls — the `cd` buys nothing and costs a prompt. Keep the command count down for the same reason: one `go get` covering both packages beats two. -- **Verify each step before building the next on top of it.** Compile the Worker before packaging it, confirm the package's target architecture before uploading, wait for the compute unit to be ready before publishing a build, and confirm the Task Queue is bound before shifting traffic. Deployment failures here surface far from their cause — an architecture or dependency mismatch appears only at first invocation, and a first-invocation failure appears as "the Worker is never invoked", several steps later. -- **When something fails, read the actual error before changing anything.** Fetch the failure reason from the provider (deployment events, logs, status fields) and fix that. Do not retry the same command with variations, and do not start editing permissions or trust policies on the theory that the problem might be access — most first-invocation failures are not permission problems, and some failures are on Temporal's side and will reproduce no matter what you change. -- **Treat the user's account as shared and pre-existing.** Assume other deployments, roles, and stacks are already there. Look before creating, extend rather than duplicate, and never delete or repurpose something you did not create without asking. When you do work around existing infrastructure — a different name, a reused role — say so explicitly in your summary rather than leaving it as a silent deviation. +- **Verify each step before building the next on top of it.** Compile the Worker before packaging it, confirm the artifact or image architecture before deploying it, wait for the compute unit to be ready before registering the build, and confirm the Task Queue is bound before shifting traffic. Deployment failures surface far from their cause: a bad Lambda package appears at first invocation, while a bad Cloud Run image appears after the pool resize when its instance tries to start. +- **When something fails, read the actual error before changing anything.** Fetch the failure reason from the provider (deployment events, logs, status fields) and fix that. Do not retry the same command with variations, and do not start editing permissions or trust policies on the theory that the problem might be access — startup failures are often in the artifact, image, or Worker configuration, and Temporal-side failures reproduce no matter what you change in the provider. +- **Treat the user's account or project as shared and pre-existing.** Assume other deployments, identities, and infrastructure state are already there. Look before creating, extend rather than duplicate, and never delete or repurpose something you did not create without asking. When you do work around existing infrastructure — a different name or a reused identity — say so explicitly in your summary rather than leaving it as a silent deviation. - **Confirm the end state from two independent signals.** A Workflow that completes in the Temporal UI *and* the Worker's own logs showing startup, Task Queue registration, and Task execution. One signal alone can mislead: a system Workflow that exists and is running proves nothing about invocation health, and a command that exits zero may have done nothing at all if it was waiting on a confirmation prompt. - **Account for what you created.** Keep the inventory as you go rather than reconstructing it at the end, say plainly that the resources are live and billable, and offer to tear them down (step 8). ## Never create or manage the WCI -Temporal creates the WCI automatically once a Worker Deployment Version has a compute provider. You never create, start, or manage it. A WCI that exists or is running is *not* evidence that invocation works — it continue-as-news and keeps running even while its Activities fail. Diagnose from Temporal's own signals: read the WCI Workflow history and look for Activity failures. Do not enumerate compute resources across regions or scan the account to reverse-engineer state. → `references/concepts.md`, `references//diagnostics.md`. +Temporal creates the WCI automatically once a Worker Deployment Version has a compute provider. You never create, start, or manage it. A WCI that exists or is running is *not* evidence that its provider action works — it continues-as-new and keeps running even while invoke or resize Activities fail. Diagnose from Temporal's own signals: read the WCI Workflow history and look for Activity failures. Do not enumerate compute resources across regions or scan the account or project to reverse-engineer state. → `references/concepts.md`, `references//diagnostics.md`. ## Provider-neutral principles @@ -189,32 +215,29 @@ Surface these early — they apply regardless of compute provider: - **A Namespace on the target cloud provider is required.** A Serverless Worker runs only on the cloud provider that hosts its Temporal Cloud Namespace — there is no cross-cloud pairing. Confirm the user has a Namespace on the provider they intend to run compute on *before* building anything; without one, the work stops there and they need either a Namespace on that provider or a different provider. A mismatch is not caught at deploy time — it fails later, at connection time. **Regions do not have to match:** a Namespace in one region can drive a compute unit in another, so never tell a user to move or re-create a Namespace to line up regions. - **Use `tcld` for every Temporal Cloud control-plane operation** — accounts, Namespaces, API keys, users, service accounts. Do not use the unified CLI's `temporal cloud …` subcommands for them. Worker Deployments and Workflows are *not* control-plane operations: they live on the Namespace frontend, have no `tcld` equivalent, and use `temporal worker deployment …`. → `references//setup.md`. - **Versioning behavior is mandatory.** Every Workflow needs `Pinned` or `AutoUpgrade`, or the Worker sets a default. -- **Deployment name and build ID must match exactly** between the Worker code and the Worker Deployment Version. A mismatch causes an invocation loop (Temporal invokes → Worker polls with the wrong version → Task not processed → invoke again). Signature: rapid repeated invocations with no Workflow progress. -- **Set the invocation deadline high enough.** Providers often default to a very short timeout. If the first invocation times out before the Worker registers the Task Queue, the binding is never created and the Worker is never invoked again. → `references//setup.md` for the exact default. +- **Deployment name and build ID must match exactly** between the Worker code and the Worker Deployment Version. The symptom follows the provider: Lambda repeatedly invokes Workers that poll under the wrong version, while Cloud Run shows running instances that poll under the wrong version and make no progress on the intended Tasks. → `references//diagnostics.md`. - **Use an immutable, versioned build per Build ID in production.** Pointing the provider at a mutable "latest" target lets code change under in-flight Workflows and cause non-determinism errors, even for Pinned Workflows. Keep a 1-to-1 mapping between each Build ID and one immutable build. → `references//versioning.md`. -- **Tune the timeout triple together for long-running Activities:** (1) worker stop timeout > longest Activity runtime, (2) shutdown deadline buffer > worker stop timeout + shutdown hook time, (3) invocation deadline > longest Activity runtime + shutdown deadline buffer. Raising one alone does not help. If the longest Activity exceeds half the maximum invocation deadline, recommend Activity Heartbeats. → `references/concepts.md`, `references//sdk-.md`. -- **Eager Activities are always disabled** — serverless invocations don't maintain persistent connections. Don't suggest them as an optimization. -- **Activities are bounded by the invocation limit** (minus the shutdown deadline buffer); Workflow duration is unbounded and can span many invocations. Flag Activities that approach the provider's limit early. → `references/concepts.md`. -- **Mixed serverless + long-lived Workers on one Task Queue:** do not enable dynamic scaling on the long-lived Workers — the two groups can't coordinate scaling and will cause unnecessary invocations. +- **Respect the compute provider's execution model, and do not assume it.** How long a Worker lives, what bounds an Activity's duration, which Worker options the provider pins, and how much timeout tuning is needed all follow from the provider's model — and they differ enough between providers that carrying an assumption across is a real source of wrong answers. Read them before advising on Activity duration, timeouts, or Worker options. → `references/concepts.md`, the selected SDK reference, and for Cloud Run `references/gcp-cloud-run/constraints.md`. +- **Mixed serverless + independently managed long-lived Workers require provider-specific treatment.** Lambda can use a fixed long-lived fleet as spillover capacity, but do not dynamically scale that fleet. Do not share a Cloud Run serverless Worker's Task Queue with another long-lived fleet: the rate-based WCI scaler sees the full queue workload and cannot subtract work handled by the other fleet, so it provisions duplicate capacity. → `references/concepts.md`, and for Cloud Run `references/gcp-cloud-run/constraints.md`. - **Secrets belong in a secret store**, not plaintext environment variables. Provider docs and quickstarts commonly pass the API key or TLS key as a plaintext environment variable; that is acceptable in a throwaway development walkthrough *only if you say so explicitly at the time*. Anything the user describes as production, shared, or long-lived gets the secret store, loaded at cold start. Either way, keep key material out of shell history and command echoes. +- **Never put a Temporal API key in `--api-key`, an inline environment assignment, generated scripts, or an echoed command.** Namespace commands inherit an already-populated `TEMPORAL_API_KEY` or use an existing Temporal CLI profile. If neither is available, pause and have the user set the variable privately in their own terminal; never ask for, print, inspect, or capture its value. This rule applies even to a one-off verification command. - **Both CLIs prompt for confirmation before mutating state, and their flags differ.** Setting the current or ramping version, and revoking an API key, all ask interactively; run non-interactively without the flag, the command exits having done nothing, which reads as success. `temporal worker deployment …` takes `--yes`; `tcld` takes the global `--auto_confirm`. Pass the right one in scripts, CI, and agent shells, and confirm the resulting state rather than trusting the exit code. → `references//setup.md`. ## Troubleshooting -Start by determining whether the Worker is being invoked at all. Then, in priority order: (1) **Validate Connection** in the Temporal UI (Workers > Deployments > select > Actions > Validate Connection) — checks credentials, role assumption, and reachability in one step; (2) check whether the version's **Task Queue is bound** — if it is, invocation and Worker startup provably work and the fault is downstream, which rules out most of the surface in one command; (3) confirm the version is **current** (CLI-created versions are not automatic, and a confirmation-prompted command may have silently done nothing); (4) check the compute provider's logs for connection, auth, or TLS errors; (5) if rapid repeated invocations show no progress, check the deployment name/build ID match. Distinguish a Temporal-side failure (reproduces no matter what you change on the provider side) from a genuine user-permission problem before editing anything. → `references//diagnostics.md`, `references/concepts.md`. +Start by identifying the provider and asking the corresponding lifecycle question: **did Lambda invoke the function, or did Cloud Run receive a pool resize and start an instance?** Then, in priority order: (1) use **Validate Connection** in the Temporal UI, interpreting only what the selected provider says it proves; (2) inspect the registration bootstrap and check whether the version's expected **Task Queue types are bound**; (3) confirm the version is **current**; (4) read the provider's logs for startup, connection, authentication, or TLS errors; and (5) check the deployment name/build ID match, using the provider-specific symptom rather than assuming an invocation loop. Distinguish a Temporal-side failure from a genuine provider-permission problem before editing anything. → `references//diagnostics.md`, `references/concepts.md`. ## Common Pitfalls High-impact mistakes — warn the user proactively. Each is a symptom → cause → fix. -1. **Deployment name / build ID mismatch → invocation loop.** *Symptom:* rapid, repeated invocations with no Workflow progress. *Cause:* the name or build ID in the Worker code doesn't match the Worker Deployment Version, so the Worker polls with the wrong version, the Task isn't processed, and Temporal invokes again. *Fix:* make the values in code exactly match the version configuration. -2. **Version not set as current.** A version created through the CLI is not automatically current; without it, Tasks don't route to the version and the Worker is never invoked. *Fix:* set it current as a separate step (the UI does this automatically). -3. **Failed first invocation.** When a version is created, the WCI invokes the Worker once to validate. If that invocation fails — missing env vars, bad TLS/auth config, missing dependencies, or an invocation deadline too short for the Worker to start and register the Task Queue — the Worker never connects, never polls, the binding is never created, and the Worker is never automatically invoked again. *Fix:* diagnose by manually invoking the compute unit, and confirm the invocation deadline is set high. -4. **Confusing the two roles.** The compute unit's execution role (grants the function permission to run) is separate from the access Temporal uses to invoke it. Never describe one as the other. → `references//iam.md`. -5. **Timeout tuning mismatch.** Raising only the shutdown deadline buffer makes the Worker stop polling earlier but gives in-flight Activities no more time; raising only the worker stop timeout doesn't make it stop polling earlier, so the provider may terminate the Worker first. *Fix:* tune the three values together (see the timeout triple above). -6. **Mutable "latest" build reference in production.** Pointing the provider at a mutable/unqualified target means the code changes on every redeploy; deploying replay-unsafe code then causes non-determinism errors for in-flight Workflows, even Pinned ones. *Fix:* publish an immutable versioned build and keep a 1-to-1 mapping between each Build ID and one build. → `references//versioning.md`. -7. **Re-creating shared permission infrastructure that already exists.** *Symptom:* the infrastructure deployment fails outright and rolls back, or it succeeds and leaves a second, redundant grant behind. *Cause:* the permission grant Temporal assumes is account-wide with a fixed default name, so a previous serverless deployment already owns it. *Fix:* check whether it exists and what owns it *before* creating; extend the existing one to cover the new Worker, and fall back to a distinctly named parallel one only when the existing infrastructure is not yours to change — saying why when you do. A failed-and-rolled-back deployment must be deleted before the name can be reused; a successful one is live infrastructure and must not be. → `references//iam.md`. -8. **Invoke permission scoped to a single build.** *Symptom:* the deployment works, then the *next* release cannot be invoked, with an error that looks like a connection or configuration problem rather than a permissions one. *Cause:* the grant named one immutable build, and the new release is a different resource. *Fix:* scope the grant to cover the base resource and all its published builds. → `references//iam.md`. +1. **Deployment name / build ID mismatch.** *Symptom:* Lambda rapidly invokes without Workflow progress; Cloud Run starts instances that look healthy but do not process the intended Tasks. *Cause:* the Worker polls under a different version from the one the WCI controls. *Fix:* make both values in code exactly match the version configuration. +2. **Version not set as current.** A version created through the CLI is not automatically current; without it, new traffic does not route to the version. *Fix:* set it current as a separate step (the UI does this automatically). +3. **Confusing the two identities.** The compute unit's own identity (which lets it run — a Lambda execution role, a Cloud Run runner service account) is separate from the identity Temporal uses to reach it. Never describe one as the other. → `references//iam.md`. +4. **Mutable "latest" build reference in production.** Pointing the provider at a mutable/unqualified target means the code changes on every redeploy; deploying replay-unsafe code then causes non-determinism errors for in-flight Workflows, even Pinned ones. *Fix:* publish an immutable versioned build and keep a 1-to-1 mapping between each Build ID and one build. → `references//versioning.md`. +5. **Re-creating shared permission infrastructure that already exists.** *Symptom:* the infrastructure deployment fails and rolls back, or succeeds but leaves a redundant grant. *Cause:* a previous serverless deployment already owns the account- or project-level role/service account under the default name. *Fix:* check ownership before creating, extend a compatible shared grant to cover the new Worker, and create a distinctly named parallel grant only when the existing infrastructure is not yours to change. A failed-and-rolled-back deployment must be deleted before its name can be reused; a successful one is live infrastructure and must not be. → `references//iam.md`. + +**Provider-specific pitfalls live with their provider.** The five above apply to any provider. For Lambda's failed-first-invocation and timeout behavior, read `references/concepts.md` and the selected AWS SDK reference. For Cloud Run scale-in behavior, read `references/gcp-cloud-run/constraints.md` and the selected Cloud Run SDK reference. ## Routing to reference files @@ -224,17 +247,18 @@ Most questions need 2–3 reference files. |---|---| | What is a Serverless Worker / the WCI? How do invocation and autoscaling work? What are the constraints? Serverless vs long-lived Workers? | `references/concepts.md` | | Deploy a Serverless Worker (happy path): write code, package, deploy, register + set-current version, verify, tear down. | `references//setup.md` + the selected `references//sdk-.md` (+ `references/concepts.md`) | -| Operator permissions and preflight; execution role vs Temporal invocation role; CloudFormation (Cloud + self-hosted). | `references//iam.md` | -| Update or redeploy; version the build, use a qualified ARN, roll back. | `references//versioning.md` (+ `references/concepts.md`) | -| Self-hosted server enablement (dynamic config, WCI, server AWS credentials). | `references//self-hosted.md` (+ `references//iam.md`) | -| Go SDK-specific options and tuned defaults, package and import, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, OpenTelemetry integration. | `references//sdk-go.md` | -| Python SDK-specific options and tuned defaults, package and import, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, OpenTelemetry integration, diagnostic signatures. | `references//sdk-python.md` | -| TypeScript SDK-specific options and tuned defaults, package and import, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, pre-bundled Workflow code, OpenTelemetry integration. | `references//sdk-typescript.md` | -| Java SDK-specific options and tuned defaults, artifact and imports, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, OpenTelemetry integration, logging and diagnostic signatures. | `references//sdk-java.md` | -| .NET SDK-specific options and tuned defaults, package and imports, API inspection, handler, RID-specific publish and packaging, runtime and deployment values, versioning-behavior configuration, connection config and `SSL_CERT_FILE`, OpenTelemetry integration, logging and diagnostic signatures. | `references//sdk-dotnet.md` | -| Add OpenTelemetry observability, Collector config, X-Ray, and IAM. | `references//observability.md` + the selected `references//sdk-.md` | -| Worker not invoked, Workflows not progressing, inspect the WCI. | `references//diagnostics.md` + the selected `references//sdk-.md` (+ `references/concepts.md`) | -| Long-running Activities and timeout relationships. Isolate Activities from resource exhaustion. | `references/concepts.md` (+ the selected `references//sdk-.md`) | +| Operator permissions and preflight; the compute unit's own identity vs the identity Temporal uses; infrastructure-as-code (CloudFormation on Lambda, Terraform on Cloud Run). | `references//iam.md` | +| Update or redeploy; make each build immutable, roll back. | `references//versioning.md` (+ `references/concepts.md`) | +| Self-hosted server enablement (dynamic config, WCI, the server's cloud credentials). | `references//self-hosted.md` (+ `references//iam.md`) | +| Go SDK-specific Worker construction and options, package and import, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, OpenTelemetry integration. | `references//sdk-go.md` | +| Python SDK-specific Worker construction and options, package and import, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, logging, OpenTelemetry integration, and diagnostic signatures. | `references//sdk-python.md` | +| TypeScript SDK-specific Worker construction and options, package and import, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, pre-bundled Workflow code, OpenTelemetry integration. | `references//sdk-typescript.md` | +| Java SDK-specific Worker construction and options, artifact and imports, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, OpenTelemetry integration, logging, and diagnostic signatures. | `references//sdk-java.md` | +| .NET SDK-specific Worker construction and options, package and imports, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, OpenTelemetry integration, logging, and diagnostic signatures. | `references//sdk-dotnet.md` | +| Add OpenTelemetry observability, collector config, tracing, and the permissions it needs. | `references//observability.md` + the selected `references//sdk-.md` | +| Worker not started, pool not resized, or Workflows not progressing; inspect the WCI. | `references//diagnostics.md` + the selected `references//sdk-.md` (+ `references/concepts.md`) | +| Long-running Activities and timeout relationships. Isolate Activities from resource exhaustion. | `references/concepts.md` + the selected `references//sdk-.md` (+ `references/gcp-cloud-run/constraints.md` for Cloud Run) | +| How long does a Worker live? What bounds an Activity? What does this provider pin or disable? What differs from another provider? | `references/concepts.md` + the selected SDK reference (+ `references/gcp-cloud-run/constraints.md` for Cloud Run) | ## Out of Scope diff --git a/references/aws-lambda/setup.md b/references/aws-lambda/setup.md index 30083bd..d4f4869 100644 --- a/references/aws-lambda/setup.md +++ b/references/aws-lambda/setup.md @@ -25,36 +25,39 @@ The selected SDK reference links its maintained sample project. Steps 4–6 and the CLI troubleshooting paths use the `temporal` CLI. Install it and authenticate it to the target Temporal Service before those steps, or commands default to `localhost:7233` and fail against Temporal Cloud. The serverless `worker deployment create-version` subcommand and its `--aws-lambda-*` flags also require a recent CLI build — see "Check the CLI version" in Step 4. -**Authenticate to Temporal Cloud (API key).** Export environment variables (the CLI and the serverless Worker packages both read these): +**Authenticate to Temporal Cloud (API key).** The CLI and the serverless Worker packages both read these environment variables. Set the non-secret values normally, but read the key privately in the user's own terminal so it never appears in shell history: ```bash export TEMPORAL_ADDRESS="..tmprl.cloud:7233" export TEMPORAL_NAMESPACE="." -export TEMPORAL_API_KEY="" +printf 'Temporal API key: ' >&2 +IFS= read -r -s TEMPORAL_API_KEY +printf '\n' >&2 +export TEMPORAL_API_KEY ``` -or configure a profile and pass `--profile prod` on each command: +A profile that already contains the key is also valid; pass `--profile prod` on each command. Configure or update only its non-secret values from the command line: ```bash temporal --profile prod config set --prop address --value "..tmprl.cloud:7233" temporal --profile prod config set --prop namespace --value "." -temporal --profile prod config set --prop api_key --value "" ``` -or configure an environment and pass `--env prod` (or set `TEMPORAL_ENV`): +An existing environment works the same way; pass `--env prod` or set `TEMPORAL_ENV`, and configure only non-secret values from the command line: ```bash temporal env set --env prod --key address --value "..tmprl.cloud:7233" temporal env set --env prod --key namespace --value "." -temporal env set --env prod --key api-key --value "" ``` -**Do not assume which of the three a user has, and do not migrate them.** `--env` (YAML, `temporal env`) is the long-standing mechanism; `--profile` (TOML, `temporal config`) is newer and the CLI still marks it EXPERIMENTAL. Both are supported — work with whichever is already configured. Read the existing values rather than asking the user to re-enter them: +Never pass the key through `config set --value`, `env set --value`, `--api-key`, or an inline assignment. If an existing profile or environment has no key, use the private environment-variable prompt above instead. Do not run the secret-reading commands through an agent shell, ask the user to paste the key into conversation, or inspect the resulting variable. + +**Do not assume which of the three a user has, and do not migrate them.** `--env` (YAML, `temporal env`) is the long-standing mechanism; `--profile` (TOML, `temporal config`) is newer and the CLI still marks it EXPERIMENTAL. Both are supported — work with whichever is already configured. Read only the non-secret values you need rather than asking the user to re-enter them: ```bash -temporal env get --env prod # --env mechanism -temporal config get --prop address # --profile mechanism +temporal env get --env prod --key address # --env mechanism +temporal --profile prod config get --prop address # --profile mechanism ``` - For Temporal Cloud the Namespace is the fully-qualified `.`, not the bare name. @@ -72,7 +75,7 @@ If this fails with an auth error, note first that this is a **frontend** call | | Control plane (accounts, Namespaces, API keys) | Namespace frontend (Workflows, Worker Deployments) | |---|---|---| | Interactive | `tcld login` | `temporal ...` with address + namespace | -| Headless | `--api-key` / `TEMPORAL_CLOUD_API_KEY` | `TEMPORAL_API_KEY` | +| Headless | `TEMPORAL_CLOUD_API_KEY` | `TEMPORAL_API_KEY` | **Use `tcld` for every Temporal Cloud control-plane operation** — accounts, Namespaces, API keys, users, service accounts. Do not use the unified CLI's `temporal cloud …` subcommands for them. @@ -85,15 +88,7 @@ Two `tcld` mechanics worth knowing before you run it in an agent shell: - `tcld login --disable-pop-up` prints the URL instead of opening a browser. Auto-open is unreliable over SSH, in containers, and in remote sessions, and the user needs the URL in the conversation either way. - `tcld` prompts for confirmation before mutating operations. Non-interactively, pass the global `--auto_confirm` (note the underscore) or set `AUTO_CONFIRM=true`, then read the resulting state back — without it the command exits clean having changed nothing. -**Go to the API key first.** It requires no CLI login, no browser handshake, and works on every account type: - -```bash -export TEMPORAL_ADDRESS="..tmprl.cloud:7233" -export TEMPORAL_NAMESPACE="." -export TEMPORAL_API_KEY="" -``` - -Have the user create the key in the Cloud UI, signing in however they normally do, and confirm the address against the endpoint shown on the Namespace page — some Namespaces have regional endpoints that do not follow the pattern above. Never ask them to paste the key into the conversation. +**Go to the API key first.** It requires no CLI login, no browser handshake, and works on every account type. Have the user create the key in the Cloud UI, signing in however they normally do, then set it with the private prompt above. Confirm the address against the endpoint shown on the Namespace page — some Namespaces have regional endpoints that do not follow the pattern above. **A control-plane login is a convenience, not a prerequisite.** When it is available it saves asking: diff --git a/references/concepts.md b/references/concepts.md index f5d141d..4fa74a6 100644 --- a/references/concepts.md +++ b/references/concepts.md @@ -6,22 +6,30 @@ **AWS Lambda — Public Preview since July 30, 2026.** Open to all Temporal Cloud customers. There is no access request, no support ticket, and no manual toggle to enable: a customer selects "AWS Lambda (Public Preview)" as the compute provider in the UI and sets up their Worker Deployment directly. Never route a user to support to "get access" for Lambda. -AWS Lambda is the only compute provider this skill supports. Do not adapt the Lambda material to any other provider. +**GCP Cloud Run — Public Preview.** Open to all Temporal Cloud customers. No access request, support ticket, or manual toggle is required. + +Those are the two supported providers. Do not adapt either one's material to a third. + +**Do not carry facts between them.** Anything about Worker lifetime, Activity duration bounds, timeouts, packaging, or tuning is provider-specific. This page names the provider whenever the action differs: Lambda is invoked; Cloud Run is resized. Public Preview is not General Availability. APIs are still evolving and may be subject to backwards-incompatible changes between versions — pin SDK and CLI versions for anything long-lived, and read the installed package's real API surface rather than writing from memory. ## What is a Serverless Worker? -A Serverless Worker is a Temporal Worker that runs on serverless compute instead of a long-lived process. -There is no always-on infrastructure to provision or scale. Temporal invokes the Worker when Tasks arrive on a Task Queue, and the Worker shuts down when the work is done. +A Serverless Worker is a Temporal Worker whose compute lifecycle is controlled by Temporal instead of by an independently operated Worker fleet. +There is no always-on compute capacity to maintain: Temporal starts capacity when needed and can return it to zero when idle. The provider resource itself—a Lambda function or Cloud Run Worker Pool—continues to exist. + +**"Starts" means different things per provider.** On AWS Lambda, Temporal invokes a function per unit of work and the Worker exits when that invocation ends. On GCP Cloud Run, Temporal resizes a pool of long-lived instances, each running an ordinary Worker that polls for its whole lifetime. Both scale to zero when idle; almost nothing else about their lifecycles is the same. -A Serverless Worker uses the same Temporal SDKs as a traditional long-lived Worker. It registers Workflows and Activities the same way. The difference is in the lifecycle: instead of the Worker starting and polling continuously, Temporal invokes the Serverless Worker on demand, the Worker starts, processes available Tasks, and then shuts down. +A Serverless Worker uses the same Temporal SDKs as a traditional long-lived Worker. It registers Workflows and Activities the same way. + +What changes is the lifecycle, and only on Lambda does it change much: instead of polling continuously, the Worker is invoked on demand, starts, processes available Tasks, and shuts down — which is why Lambda needs a dedicated serverless Worker package (`aws-lambda/sdk-.md`). **On Cloud Run the Worker code is unchanged from a long-lived Worker**; the only addition is Worker Versioning, and there is no Cloud Run Worker package at all. Serverless Workers require Worker Versioning. Each Serverless Worker must be associated with a Worker Deployment Version that has a compute provider configured. Each Workflow must have an `AutoUpgrade` or `Pinned` versioning behavior, set per-Workflow or as a Worker-level default. -## How Serverless invocation works +## How Temporal controls Serverless Workers With long-lived Workers, the Worker process starts, connects to Temporal, and polls a Task Queue for work. Temporal does not need to know anything about the Worker's infrastructure. @@ -32,7 +40,7 @@ With Serverless Workers, Temporal starts the Worker. One WCI Workflow runs per Worker Deployment Version that has a compute provider configured. The WCI runs in the same Namespace as your Worker Deployment. -The WCI responds to two triggers: sync match failures and Task Queue backlog. When either trigger fires, the WCI produces a scaling action, such as invoking the configured compute provider (for example, calling AWS Lambda's `InvokeFunction` API) to start new Workers. +The WCI responds to sync match failures and periodically reads Task Queue metrics. It turns those inputs into an action compatible with the provider: invoke a Lambda function, or update a Cloud Run Worker Pool's manual instance count. You can list WCI Workflows in your Namespace: @@ -54,43 +62,41 @@ temporal workflow show \ ``` -### Invocation flow +### Shared Task routing -The invocation flow works as follows: +1. A Task is submitted, for example by `StartWorkflow` or `ScheduleActivity`. +2. Matching attempts to route it directly to an available Worker in a sync match. +3. If no Worker is available, Matching adds the Task to the backlog and signals the WCI for that Worker Deployment Version. +4. The WCI asks the version's scaling algorithm for an action and applies it through the configured provider. -1. A Task is submitted (for example, `StartWorkflow` or `ScheduleActivity`). -2. The Matching Service attempts to route the Task directly to an available Worker (a sync match). -3. If a Worker is available, the Task is routed to that Worker. -4. If no Worker is available (sync match fails), the Matching Service pushes a signal to the WCI, and the WCI invokes the configured compute provider. -5. The Serverless Worker starts, creates a Temporal Client, and begins polling the Task Queue. -6. The Worker processes available Tasks until it exits (see Worker lifecycle). +After that point the providers diverge: -Each invocation is independent. The Worker creates a fresh client connection on every invocation. There is no connection reuse or shared state across invocations. +- **AWS Lambda:** the WCI invokes the function. Each invocation creates a Worker and client connection, processes available Tasks, and shuts down. Invocations do not share a connection or in-memory state. +- **GCP Cloud Run:** the WCI increases the Worker Pool's manual instance count. Cloud Run starts an instance whose ordinary Worker connects once and polls for the instance's lifetime. The WCI later lowers the count as demand drains. ## Autoscaling -The WCI automatically scales Serverless Workers based on Task Queue signals. When Tasks arrive and no Worker is available, the WCI invokes new Workers. When the Tasks are done, Workers exit and scale to zero. - -The WCI uses two signals to decide when to invoke new Workers: +The WCI automatically scales Serverless Workers from Task Queue signals and metrics. The resulting action depends on the provider. ### Sync match failure -When a Task is submitted, the Matching Service attempts to route it directly to an available Worker. If no Worker is available, the sync match fails, and the Matching Service pushes a signal to the WCI. The WCI then invokes a new Worker. This is the primary scaling path. +When a Task is submitted, Matching attempts to route it directly to an available Worker. If no Worker is available, the sync match fails and Matching signals the WCI. Lambda's no-sync algorithm can invoke another function; Cloud Run's rate-based algorithm can immediately increase the planned pool size, subject to its cooldown and maximum. Because the Matching Service pushes match failures to the WCI as they happen rather than the WCI polling on a timer, latency stays low and scaling is responsive. ### Task Queue backlog -The WCI monitors Task Queue metadata to determine whether pending Tasks exist without enough Workers to process them. If there are Tasks on the queue and not enough Workers, the WCI invokes additional Workers. +The WCI periodically reads version-level Task Queue arrival rate, dispatch rate, and backlog. Cloud Run's rate-based algorithm uses these metrics to calculate a desired instance count and explicitly resizes the pool; the periodic path also scales the pool down, including to zero. Lambda Workers instead end with their invocations and do not use this worker-set sizing model. ## Scaling with long-lived Workers -Serverless Workers can share a Task Queue with long-lived Workers. Because Serverless Workers are only invoked on sync match failure, Serverless Workers only pick up Tasks that no long-lived Worker was available to handle. In practice, the Serverless Workers act as spillover capacity for the long-lived fleet. - -**Warning:** If you configure Serverless and long-lived Workers on the same Task Queue, do not enable dynamic scaling on the long-lived Workers. The two groups cannot coordinate their scaling behavior. If both scale dynamically, the long-lived Workers may scale up to handle the same Tasks that Temporal is simultaneously invoking Serverless Workers for, leading to unnecessary invocations and unpredictable scaling. +- **AWS Lambda:** a Lambda Worker can share a Task Queue with a fixed long-lived fleet and act as spillover when sync matching finds no available poller. Do not dynamically scale the long-lived fleet as well; the two scaling systems cannot coordinate. +- **GCP Cloud Run:** use a separate Task Queue from any independently managed long-lived fleet. The rate-based WCI scaler reads the full version-level arrival, dispatch, and backlog metrics and cannot subtract work handled by the other fleet, so sharing provisions duplicate capacity. → `gcp-cloud-run/constraints.md`. ## Worker lifecycle +**This section describes providers that invoke per unit of work, such as AWS Lambda.** On GCP Cloud Run, an instance connects once and polls for its whole lifetime: there are no per-invocation phases, and none of the tuning below applies. → `gcp-cloud-run/constraints.md`. + A single Serverless Worker invocation has three phases: init, work, and shutdown. ### Init phase @@ -107,52 +113,31 @@ The Worker stops polling, waits for in-flight Tasks to finish, and runs any shut ### Tuning for long-running Activities -If your Worker handles long-running Activities, set these three values together: - -- **Worker stop timeout > longest Activity runtime.** Gives in-flight Activities enough time to finish after polling stops. -- **Shutdown deadline buffer > Worker stop timeout + shutdown hook time.** Ensures the drain and any shutdown hooks complete before the compute provider terminates the environment. -- **Invocation deadline > longest Activity runtime + shutdown deadline buffer.** Set on the compute provider to give each invocation enough total runtime. - -If your longest-running Activity runs longer than half the maximum invocation deadline, use Activity Heartbeats to record the state of the Activity execution so that the next retry can pick up where it left off. - -Example: if your longest Activity runtime is 5 minutes, and your shutdown hooks take 3 seconds, set the Worker stop timeout to more than 5 minutes, and the shutdown deadline buffer to more than 303 seconds (5 minutes + 3 seconds). Set your invocation deadline to at least 10 minutes and 3 seconds. +For an invocation-based Worker, tune three values together: -The Worker stop timeout controls how long the Worker waits for in-flight Tasks to finish after it stops polling. The shutdown deadline buffer controls how much time before the invocation deadline the Worker stops polling for Tasks. +- **Worker stop timeout > longest Activity runtime.** This gives in-flight Activities time to finish after polling stops. +- **Shutdown deadline buffer > Worker stop timeout + shutdown hook time.** This leaves time for draining and shutdown hooks before the provider terminates the environment. +- **Invocation deadline > longest Activity runtime + shutdown deadline buffer.** This gives the invocation enough total runtime. -Raising only the shutdown deadline buffer makes the Worker stop polling earlier, but does not give in-flight Tasks any more time to complete. +Raising one alone does not help. Raising only the shutdown deadline buffer stops polling earlier without giving in-flight Tasks more time; raising only the Worker stop timeout does not make polling stop earlier, so the provider can terminate the Worker before drainage finishes. -Raising only the Worker stop timeout does not make the Worker stop polling earlier, which means the compute provider might terminate the Worker before the full stop timeout completes. +If the longest Activity exceeds half the provider's maximum invocation deadline, use Activity Heartbeats so a retry can resume from recorded progress. For example, a five-minute Activity and three seconds of shutdown hooks require a Worker stop timeout above five minutes, a shutdown deadline buffer above 303 seconds, and an invocation deadline of at least ten minutes and three seconds. ## Failure handling Serverless Workers rely on Temporal's standard retry and timeout semantics to recover from failures. -### Worker crash +### Worker crash or instance termination -If a Worker invocation crashes (out of memory, unhandled exception, etc.): +If a Worker crashes or its compute is terminated, the in-flight Task is not acknowledged. Temporal applies the configured timeout and retry policy, and another Worker can receive the retry. On Lambda that means another invocation; on Cloud Run it means another running or replacement pool instance. Activity Heartbeats preserve progress for long-running work that can be interrupted. -- The Activity Timeout fires after the configured duration. -- Temporal retries the Activity on a different Worker invocation. -- No manual intervention is required. +### Provider capacity limit -### Provider concurrency limit - -If the compute provider's concurrency limit is reached (for example, AWS Lambda account concurrency): - -- Further invocations from the WCI fail. -- Tasks remain in the Task Queue backlog. No data loss occurs. -- Processing slows until concurrency frees up. +If the provider cannot add capacity—Lambda account concurrency, Cloud Run regional quotas, or the configured Cloud Run `max_count`—Tasks remain in the Task Queue backlog without data loss and processing slows until capacity becomes available. The provider-side symptom differs: Lambda invocations are throttled or rejected, while a Cloud Run pool stops growing or its update fails. ### Resource exhaustion across Activity slots -By default, a single Worker invocation may run multiple Activity slots. A crash or resource exhaustion in one Activity can affect other Activities running in the same invocation. - -To isolate Activities from each other: - -- Split Workflow and Activity Workers into separate compute functions. -- Set Activity slots to 1 per invocation. - -With single-slot configuration, each Activity gets a dedicated execution environment. +A Worker process may run multiple Activity slots, so a crash or resource exhaustion in one Activity can affect other Activities in that same process. On Lambda, split Workflow and Activity Workers into separate functions or use one Activity slot per invocation for execution-environment isolation. On Cloud Run, size instance resources and Worker concurrency together; one slot limits concurrency within an instance but does not turn the long-lived instance into a per-Activity environment. → the selected SDK reference and `gcp-cloud-run/constraints.md` for Cloud Run. ## Constraints @@ -160,39 +145,27 @@ With single-slot configuration, each Activity gets a dedicated execution environ | Constraint | Detail | |---|---| -| Activity duration | Must complete within the compute provider's invocation limit (minus shutdown deadline buffer). For AWS Lambda, the maximum is 15 minutes. | -| Workflow duration | No limit. Workflows of any duration work, regardless of the invocation timeout. A Workflow runs across as many invocations as needed. | -| Worker code | Same Temporal SDK Worker code, using the serverless Worker package for your SDK. | +| Activity duration | On Lambda, must complete within its invocation limit minus the shutdown deadline buffer; the invocation ceiling is 15 minutes. Cloud Run imposes no per-invocation ceiling, but scale-in can interrupt an Activity. → the selected SDK reference and `gcp-cloud-run/constraints.md` for Cloud Run. | +| Workflow duration | No compute-provider limit. Workflow state is durable in Temporal and does not depend on one Lambda invocation or Cloud Run instance remaining alive. | +| Worker code | Same Temporal SDK Worker code. On Lambda it runs through that SDK's serverless Worker package; on Cloud Run it is an ordinary long-lived Worker with no extra package. | | Versioning | Worker Versioning is required. Each Workflow must have an `AutoUpgrade` or `Pinned` behavior, set per-Workflow or as a Worker-level default. | ## Worker Versioning with Serverless Workers -Serverless Workers require Worker Versioning, and the compute provider must invoke a stable, immutable build for each Worker Deployment Version. With AWS Lambda, this means aligning two versioning systems: +Serverless Workers require Worker Versioning, and the compute provider must target a **stable, immutable build** for each Worker Deployment Version. That means aligning two versioning systems: - **Temporal Worker Deployment Versions** — identified by deployment name and Build ID. Each Workflow runs against a specific Worker Deployment Version (Pinned) or moves between them on routing changes (Auto-Upgrade). -- **AWS Lambda function versions** — immutable numbered snapshots of your Lambda function code (`1`, `2`, `3`, ...). - -For production workloads, map each Worker Deployment Version to exactly one Lambda function version, and configure the compute provider with the qualified versioned ARN for that Lambda version (for example, `arn:aws:lambda:us-east-1:123:function:my-worker:5`). - -For development or non-critical workloads, you can use an unqualified ARN to iterate without publishing a new Lambda function version each time. - -**Caution:** An unqualified ARN (no version suffix) points at `$LATEST`, which changes on every redeploy. Without a versioned ARN, deploying replay-unsafe code causes non-determinism errors for in-flight Workflows, even for Workflows annotated as Pinned. - -The choice of Pinned or Auto-Upgrade controls how Workflows move between Worker Deployment Versions in Temporal. It does not change how a Worker Deployment Version targets Lambda. Both behaviors expect a versioned ARN that points at one immutable Lambda function version. +- **The provider's own unit of immutability** — a published Lambda function version, pinned by a qualified ARN; or on Cloud Run a dedicated Worker Pool per Build ID, because the compute configuration names a pool and not a revision. Keep a one-to-one mapping between it and the Build ID. -| Versioning Behavior | With versioned Lambda ARN | Without versioned Lambda ARN | -|---|---|---| -| **Pinned** | Existing Workflows stay on their original Lambda function version until they complete. | Existing Workflows stay on their original Worker Deployment Version, but the underlying Lambda code has already changed since `$LATEST` updated at redeploy. The new code must be replay-compatible. | -| **Auto-Upgrade** | Existing Workflows move to the new Worker Deployment Version and its new Lambda function version at the next Workflow Task after you move the Current Version. | The Lambda redeploy already changed the code for all versions. Setting the Current Version only changes routing, not which code runs. | - +**Pointing a Worker Deployment Version at a mutable target causes non-determinism errors for in-flight Workflows, including Pinned ones.** Pinned routes Workflows to a version; it cannot pin code that changed underneath that version. The failure is the same on both providers but is reached differently — on Lambda you have to choose it by registering an unqualified ARN, while on Cloud Run a plain redeploy into a live pool does it — so read the provider file for which action is the dangerous one. → `aws-lambda/versioning.md`, `gcp-cloud-run/versioning.md`. -See `aws-lambda/versioning.md` for the step-by-step `aws lambda publish-version` workflow and `aws-lambda/setup.md` (Step 4) for how to configure the compute provider with a versioned ARN. +Pinned or Auto-Upgrade controls how Workflows move between Worker Deployment Versions in Temporal. It does not change how a Worker Deployment Version targets the provider; both behaviors expect one immutable build per version. ## Compute providers -A compute provider is the configuration that tells Temporal how to invoke a Serverless Worker. The compute provider is set on a Worker Deployment Version and specifies the provider type, the invocation target, and the credentials Temporal needs to trigger the invocation. +A compute provider is the configuration that tells Temporal how to control Serverless Worker capacity. It is set on a Worker Deployment Version and specifies the provider type, compute target, and credentials Temporal needs to invoke a function or resize a worker set. -For example, an AWS Lambda compute provider includes the Lambda function ARN and the IAM role that Temporal assumes to invoke the function. +For example, an AWS Lambda compute provider includes the Lambda function ARN and the IAM role that Temporal assumes to invoke the function; a Cloud Run compute provider names the project, region, and Worker Pool, plus the service account Temporal impersonates to scale it. Compute providers are only needed for Serverless Workers. Traditional long-lived Workers do not require a compute provider because the Worker process lifecycle is not managed by the Temporal server. @@ -203,15 +176,16 @@ Compute providers are only needed for Serverless Workers. Traditional long-lived | Provider | Description | |---|---| | AWS Lambda | Temporal assumes an IAM role in your AWS account to invoke a Lambda function. | +| GCP Cloud Run | Temporal impersonates a service account in your Google Cloud project to scale a Worker Pool. | ## Why use Serverless Workers? -- **Reduce operational overhead.** No always-on infrastructure to manage and no autoscaling policies to tune. Temporal and the compute provider handle invocation and scaling. -- **Get started faster.** Deploying a Worker is as simple as deploying a function. No Kubernetes, container orchestration, or scaling strategy required. -- **Scale automatically.** The compute provider handles scaling natively. When traffic drops, instances scale down. When there is no work, there is no compute running. -- **Pay only for what you use.** Workers run only when Tasks are available. For low or intermittent volume workloads, this pay-per-invocation model can significantly reduce compute costs. +- **Reduce operational overhead.** Temporal drives capacity from Task Queue demand: invoking Lambda or setting the Cloud Run Worker Pool size. +- **Avoid managing a continuously provisioned fleet.** Deploy a function or a Worker Pool without building a separate autoscaling control plane. +- **Scale automatically.** Capacity grows with demand and can return to zero when idle. +- **Pay only while compute runs.** Lambda bills invocations; Cloud Run bills running pool instances. Both can reduce idle cost for low or intermittent workloads. ## When to use Serverless Workers @@ -229,16 +203,17 @@ May not be ideal when: -- Activities are long-running and cannot be interrupted. AWS Lambda has a 15-minute execution limit. Activities that run longer and cannot be broken into smaller steps need a different hosting strategy or a provider with longer limits. +- Activities are long-running and cannot be interrupted, on a provider with a per-invocation ceiling — Lambda's is 15 minutes. Activities that run longer and cannot be broken into smaller steps need a different hosting strategy, or a provider without that ceiling. - Workloads require sustained high throughput. Long-lived Workers on dedicated compute may be more cost-effective and performant. -- You need persistent connections. Some features require a persistent connection between the Worker and Temporal, which serverless invocations do not maintain. +- You need persistent connections and the provider invokes per unit of work. Some features require a persistent connection between the Worker and Temporal, which per-invocation Workers do not maintain; a pool-based provider holds one for the instance's lifetime. ## How Serverless Workers compare to long-lived Workers -| | Long-lived Worker | Serverless Worker | -|---|---|---| -| **Lifecycle** | Long-lived process that runs continuously. | Invoked on demand. Starts and stops per invocation. | -| **Scaling** | You manage scaling (Kubernetes HPA, instance count, etc.). | Temporal invokes additional instances as needed, within the compute provider's concurrency limits. | -| **Connection** | Persistent connection to Temporal. | Fresh connection on each invocation. | +| | Independently managed Worker | AWS Lambda Serverless Worker | GCP Cloud Run Serverless Worker | +|---|---|---|---| +| **Lifecycle** | Long-lived process; you decide when it runs. | Short-lived Worker created for each function invocation. | Ordinary long-lived Worker inside each pool instance; WCI controls the instance count. | +| **Scaling** | You manage replicas or instances. | WCI invokes functions after Task Queue signals. | WCI explicitly resizes the Worker Pool from signals and periodic queue metrics. | +| **Connection** | Persistent for the process lifetime. | Fresh connection per invocation. | Persistent for the pool instance lifetime. | +| **Scale to zero** | Only if you build and operate it. | Invocations end when work drains. | WCI lowers the pool's manual instance count to zero. | diff --git a/references/gcp-cloud-run/setup.md b/references/gcp-cloud-run/setup.md index 7651d90..5981207 100644 --- a/references/gcp-cloud-run/setup.md +++ b/references/gcp-cloud-run/setup.md @@ -4,7 +4,7 @@ End-to-end: write a standard Worker, containerize it, push the image, create a W ## Prerequisites -- **Cloud Run support is Pre-release and access-gated.** The user creates a support ticket or contacts their account team. Confirm this before anything else. +- **Cloud Run support is Public Preview and open to all Temporal Cloud customers.** No access request is required. - A Temporal Cloud account with a **GCP-hosted Namespace**, or self-hosted Temporal Service v1.31.0+. The Namespace must be hosted on GCP; its region need not match the pool's. - For self-hosted, complete `self-hosted.md` first. - Every Workflow must declare a versioning behavior, or the Worker must set a default.