diff --git a/AGENTS.md b/AGENTS.md index 2315542..efe19e9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -110,7 +110,7 @@ The four brainstore configmaps (`brainstore-reader-configmap.yaml`, `brainstore- ### Brainstore Automation Writer loop config -The optional Automation Writer pool (`brainstore.automationwriter`, default `replicas: 0`) isolates the automations writer loop via `BRAINSTORE_WRITER_LOOP_CONFIG`. That variable exists starting in brainstore image `v2.16.0`. Enabling the pool (`replicas > 0`) fails the render when `brainstore.image.tag` is a release tag (`vX.Y.Z`) older than `v2.16.0`. Commit SHAs and other custom tags must keep rendering, so `braintrust.automationWriter.validate` only calls `semverCompare` after the tag matches the release-tag regex (`semverCompare` errors on a SHA). Do not bump the chart-wide default image for this. `brainstore-automationwriter-configmap.yaml` sets `include:automations`, and `brainstore-writer-configmap.yaml` sets `exclude:automations` only when `brainstore.automationwriter.replicas > 0`. A null or missing replica count is 0: an empty `replicas` field would make Kubernetes run 1 pod while the writer still handled automations. A missing `brainstore.automationwriter` map — what `helm upgrade --reuse-values` retains from a chart that predates the pool — is disabled: the writer does not exclude automations, and the automation writer manifests are omitted. A partial map, such as `--set brainstore.automationwriter.replicas=1` on those retained values, is merged with the template defaults in `braintrust.automationWriter.defaults` before any field is read. On Azure with the Container Storage driver, `brainstore.automationwriter.volume.size` is required only when `replicas > 0`; a zero-replica pool renders with `emptyDir` so an upgrade does not demand disk for a pool that is off. These two must stay in sync: the automation writer runs automations and the regular writer must exclude them whenever the pool is enabled. An enabled pool has an internal Service and publishes `BRAINSTORE_AUTOMATION_WRITER_URL` to all API pools whenever replicas are positive. A compatible API release decides which queries use the endpoint. The API startup gate and its RBAC deployment names cover readers, writers, and fast readers only. Resolve names and Service fields through `braintrust.automationWriter.config` so partial reused values keep working. Use `brainstore.automationwriter.service.port` wherever an automation writer port is required. Coordinate API routing changes before scaling the pool to zero; old API processes retain their URL until restarted. +The optional Automation Writer pool (`brainstore.automationwriter`, default `replicas: 0`) isolates the automations writer loop via `BRAINSTORE_WRITER_LOOP_CONFIG`. That variable exists starting in brainstore image `v2.16.0`. Enabling the pool (`replicas > 0`) fails the render when `brainstore.image.tag` is a release tag (`vX.Y.Z`) older than `v2.16.0`. Commit SHAs and other custom tags must keep rendering, so `braintrust.automationWriter.validate` only calls `semverCompare` after the tag matches the release-tag regex (`semverCompare` errors on a SHA). Do not bump the chart-wide default image for this. `brainstore-automationwriter-configmap.yaml` sets `include:automations`, and `brainstore-writer-configmap.yaml` sets `exclude:automations` only when `brainstore.automationwriter.replicas > 0`. A null or missing replica count is 0: an empty `replicas` field would make Kubernetes run 1 pod while the writer still handled automations. A missing `brainstore.automationwriter` map — what `helm upgrade --reuse-values` retains from a chart that predates the pool — is disabled: the writer does not exclude automations, and the automation writer manifests are omitted. A partial map, such as `--set brainstore.automationwriter.replicas=1` on those retained values, is merged with the template defaults in `braintrust.automationWriter.defaults` before any field is read. On Azure with the Container Storage driver, `brainstore.automationwriter.volume.size` is required only when `replicas > 0`; a zero-replica pool renders with `emptyDir` so an upgrade does not demand disk for a pool that is off. These two must stay in sync: the automation writer runs automations and the regular writer must exclude them whenever the pool is enabled. An enabled pool has an internal Service and publishes `BRAINSTORE_AUTOMATION_WRITER_URL` to all API pools whenever replicas are positive. A compatible API release decides which queries use the endpoint. Deployment ordering and its RBAC must cover all rendered Brainstore roles: readers, writers, fast readers, and automation writers. Include zero-replica Deployments until their old pods have drained; omit automation writers only when their Deployment is not rendered. Resolve names and Service fields through `braintrust.automationWriter.config` so partial reused values keep working. Use `brainstore.automationwriter.service.port` wherever an automation writer port is required. Coordinate API routing changes before scaling the pool to zero; old API processes retain their URL until restarted. ### Disruption budgets and GKE Standard diff --git a/braintrust/README.md b/braintrust/README.md index cf9c0a6..29bb31e 100644 --- a/braintrust/README.md +++ b/braintrust/README.md @@ -475,7 +475,8 @@ The API receives `BRAINSTORE_AUTOMATION_WRITER_URL` whenever the pool has replicas. A compatible API release decides which queries use this endpoint. With no replicas, the URL is omitted and the API falls back to the regular writers. -The API startup gate covers readers, writers, and fast readers. +The API startup gate covers every rendered reader and writer role, including +automation writers and pools that are scaling down to zero. The include/exclude split follows `brainstore.automationwriter.replicas` in the Helm values. `kubectl scale` or an autoscaler changes the live pod count only; @@ -554,3 +555,47 @@ Example values files for different cloud providers and configurations are locate - `examples/google-autopilot-cel/values.yaml`: GKE Autopilot deployment with CEL-friendly security settings. - `examples/google-api-isolation-autoscaling/values.yaml`: Minimal example for API workload isolation and per-pool autoscaling on GKE (combine with an Autopilot or Standard values file). - `examples/google-standard/values.yaml`: GKE Standard deployment. + +## Release ordering + +`deploymentOrdering.enabled` (default `true`) orders startup as migrations → +Brainstore → API pools, enabled AI Gateway, and enabled Loop Runtime. Only changed +workload pod templates roll out; unchanged workloads keep serving. A no-op Helm +upgrade creates no new migration Job or pod rollout. + +**Upgrade:** `api.brainstoreStartupGate` is no longer read. Existing opt-outs must +set `deploymentOrdering.enabled: false`; the old `false` value does not disable +the new ordering. + +Gateway has its own Kubernetes ServiceAccount regardless of ordering; configure +its name with `aiGateway.serviceAccount.name`. + +Ordering gates container startup, not Kubernetes object updates. Gates read the +latest desired state from a stable ConfigMap and verify each Brainstore role's +expected pod-template identity, observed generation, and complete readiness. +Old Ready pods cannot satisfy a changed role's gate; unchanged healthy roles can. + +`deploymentOrdering.timeoutSeconds` bounds each phase. Stable tags select a +migration bundle covering the newest API pool or Brainstore release. Opaque +API/Brainstore pins require both a stable `minimumVersion` compatibility assertion +and an explicit standalone-API `migrationImage` containing all required migrations. +Known stable images still undergo version checks; opaque Brainstore pods must +match their Deployment's exact image and declared release. See [values.yaml](values.yaml). +Use `--wait --wait-for-jobs` with a sufficient Helm timeout. + +Migration Jobs use `-migrate-`, keyed by release identity, +image, and execution configuration. Successful Jobs and immutable configuration +snapshots are retained and reused, including on rollback. After diagnosing a failed +migration, delete its failed Job before +retrying the same inputs with Helm. Rollback does not reverse SQL changes; +migrations must remain backward-compatible. External Secret value changes and +replaced images behind unchanged tags are not detected; use immutable image pins +and explicitly restart affected workloads when changing external secrets. + +The legacy startup-gate ConfigMap is preserved unchanged while old API replicas +may still need it; new pods use the stable ordering ConfigMap. + +After uninstall, before reusing a release name, delete Jobs labeled +`braintrust.dev/migration-release=` with foreground deletion and wait for +completion, then delete labeled ConfigMaps, Roles, and RoleBindings. Never delete +records still needed by active workloads or rollback revisions. diff --git a/braintrust/files/brainstore-startup-gate.py b/braintrust/files/brainstore-startup-gate.py index cf4ff72..6d42898 100644 --- a/braintrust/files/brainstore-startup-gate.py +++ b/braintrust/files/brainstore-startup-gate.py @@ -1,8 +1,8 @@ -"""Wait for compatible Brainstore capacity before starting an API container. +"""Wait for compatible Brainstore rollouts before application startup. Uses only Python's standard library and read-only Kubernetes API requests. -The required version is fixed in the API Pod's environment, not a ConfigMap that -changes underneath an older API Pod when the next release is installed. +The required version is fixed in the application's Pod environment. The expected +fleet is read from the stable ordering ConfigMap on every observation. """ import json @@ -19,6 +19,8 @@ VERSION = re.compile(r"v?(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\Z") TERMINAL_PHASES = {"Failed", "Succeeded"} +TEMPLATE_ANNOTATION = "braintrust.dev/pod-template" +TEMPLATE_HASH = re.compile(r"[0-9a-f]{64}\Z") def release_version(value): @@ -43,7 +45,7 @@ def compatible(image, minimum): return False # This assumes Brainstore preserves compatibility with older APIs within # a major release; version numbers alone cannot establish that contract. - # Unknown tags and major-version migrations must not pass this gate. + # Unknown tags cannot establish compatibility without a template assertion. return version[0] == minimum[0] and version >= minimum @@ -76,47 +78,92 @@ def ready(pod, container): ) -def check_deployment(deployment, pods, container, minimum): - """Return a wait reason, or None when every potential backend is compatible. +def check_deployment(deployment, pods, target, minimum): + """Return a wait reason, or None when the compatible rollout is complete. - We deliberately do not require an exact image or a finished *compatible* - rollout: older APIs must be able to restart while a later Brainstore release - rolls. Incompatible terminating Pods still block, because their connections - may remain usable until the process exits. + The current Deployment rollout must finish, even for an older application + restarting during a later compatible release. All live Service backends + must belong to that rollout, and terminating Pods must disappear because + connections may remain usable until their process exits. - Deployment rollouts replace Pods, so use each Pod's declared release tag. - Container status can report a different alias or only a digest. Neither is - a portable release identifier; manual in-place Pod image changes are outside - this check's contract. + Stable image tags establish compatibility directly. Opaque image pins + instead require the template's declared version and exact image/annotation + matches on every live Pod. Container status may report only an alias or + digest; manual in-place Pod image changes are outside this check's contract. """ metadata = deployment.get("metadata", {}) spec = deployment.get("spec", {}) status = deployment.get("status", {}) name = metadata.get("name", "unknown") + container = target["container"] if metadata.get("deletionTimestamp"): return f"{name}: deployment is being deleted" # Do not trust availability status from before the latest spec update. - if status.get("observedGeneration", 0) < metadata.get("generation", 1): + generation = metadata.get("generation") + observed = status.get("observedGeneration", 0) + if (type(generation) is not int or generation <= 0 + or type(observed) is not int or observed < generation): return f"{name}: controller has not observed the desired generation" - desired = spec.get("replicas", 1) + desired = target["replicas"] + if type(spec.get("replicas", 1)) is not int or spec.get("replicas", 1) != desired: + return f"{name}: waiting for the expected {desired} replicas in the Deployment spec" + min_ready = target["minReadySeconds"] + if (type(spec.get("minReadySeconds", 0)) is not int + or spec.get("minReadySeconds", 0) != min_ready): + return f"{name}: waiting for the expected minReadySeconds in the Deployment spec" + template = spec.get("template", {}) + template_annotations = template.get("metadata", {}).get("annotations", {}) + if template_annotations.get(TEMPLATE_ANNOTATION) != target["template"]: + return f"{name}: waiting for the expected Pod template in the Deployment spec" live = [pod for pod in pods if pod.get("status", {}).get("phase") not in TERMINAL_PHASES] + opaque_image = None if desired == 0: - return f"{name}: waiting for scaled-down Pods to disappear" if live else None - # Check desired state too: compatible Pods are insufficient if the - # Deployment is about to replace them with an incompatible image. - image = container_image(spec.get("template", {}).get("spec", {}), container) - if not compatible(image, minimum): - return f"{name}: desired image {image!r} is below or outside the required release" + if live: + return f"{name}: waiting for scaled-down Pods to disappear" + else: + # An assertion must never override a known incompatible stable image. + desired_image = container_image(template.get("spec", {}), container) + if not desired_image: + return f"{name}: desired Brainstore image is missing" + try: + image_version(desired_image) + except ValueError: + version_key = "braintrust.dev/brainstore-version" + declared_version = template_annotations.get(version_key) + try: + version = release_version(declared_version) + except (TypeError, ValueError): + return f"{name}: opaque desired image requires a stable declared Brainstore version" + if version[0] != minimum[0] or version < minimum: + return f"{name}: declared Brainstore version is below or outside the required release" + opaque_image = desired_image + else: + if not compatible(desired_image, minimum): + return f"{name}: desired image {desired_image!r} is below or outside the required release" for pod in live: image = container_image(pod.get("spec", {}), container) - if not compatible(image, minimum): - pod_name = pod.get("metadata", {}).get("name", "unknown") + pod_metadata = pod.get("metadata", {}) + if opaque_image is not None: + if image != opaque_image: + return f"{name}: waiting for Pods with the exact desired opaque image" + if pod_metadata.get("annotations", {}).get(version_key) != declared_version: + return f"{name}: waiting for Pods with the declared Brainstore version" + elif not compatible(image, minimum): + pod_name = pod_metadata.get("name", "unknown") return f"{name}: incompatible Pod {pod_name} still exists ({image!r})" + if pod_metadata.get("annotations", {}).get(TEMPLATE_ANNOTATION) != target["template"]: + return f"{name}: waiting for Pods with the expected Pod template" + if pod_metadata.get("deletionTimestamp"): + return f"{name}: waiting for terminating Pods to disappear" # Require both current Pod readiness and Deployment availability; the # latter also accounts for the Deployment's minReadySeconds setting. ready_count = sum(ready(pod, container) for pod in live) - if ready_count < desired or status.get("availableReplicas", 0) < desired: - return f"{name}: waiting for {desired} available compatible Pods ({ready_count} ready)" + counters = ("updatedReplicas", "readyReplicas", "availableReplicas", "replicas") + if any(type(status.get(counter, 0)) is not int or status.get(counter, 0) != desired + for counter in counters): + return f"{name}: waiting for all {desired} replicas to complete the current rollout" + if len(live) != desired or ready_count != desired: + return f"{name}: waiting for exactly {desired} current Ready Pods" return None @@ -175,29 +222,93 @@ def pods(self, selector, deadline): return items +def load_ordering_config(client, name, deadline): + name = urllib.parse.quote(name, safe="") + resource = client.get( + f"/api/v1/namespaces/{client.namespace}/configmaps/{name}", deadline, + ) + if not isinstance(resource, dict) or not isinstance(resource.get("data"), dict): + raise ValueError("ordering ConfigMap data is missing") + config = json.loads(resource["data"]["ordering.json"]) + if not isinstance(config, dict): + raise ValueError("ordering configuration must be an object") + if not isinstance(config.get("job"), str) or not config["job"]: + raise ValueError("ordering configuration requires a Job name") + targets = config.get("targets") + if not isinstance(targets, list) or not targets: + raise ValueError("ordering configuration requires nonempty targets") + for target in targets: + if not isinstance(target, dict): + raise ValueError("ordering targets must be objects") + for field in ("deployment", "container"): + if not isinstance(target.get(field), str) or not target[field]: + raise ValueError(f"ordering target requires a {field} name") + fingerprint = target.get("template") + if not isinstance(fingerprint, str) or not TEMPLATE_HASH.fullmatch(fingerprint): + raise ValueError("ordering target requires a SHA256 Pod template") + for field in ("replicas", "minReadySeconds"): + if type(target.get(field)) is not int or target[field] < 0: + raise ValueError(f"ordering target requires nonnegative integer {field}") + return config + + +def job_complete(job): + if not isinstance(job, dict): + raise ValueError("Job observation must be an object") + metadata = job.get("metadata", {}) + status = job.get("status", {}) + if not isinstance(metadata, dict) or not isinstance(status, dict): + raise ValueError("Job metadata and status must be objects") + conditions = status.get("conditions", []) + if not isinstance(conditions, list) or any(not isinstance(c, dict) for c in conditions): + raise ValueError("Job conditions must be objects") + if any(c.get("type") in {"Failed", "FailureTarget"} and c.get("status") == "True" + for c in conditions): + raise RuntimeError("Migration Job failed; inspect its logs before retrying") + if metadata.get("deletionTimestamp"): + return False + return any(c.get("type") == "Complete" and c.get("status") == "True" for c in conditions) + + +def migration_complete(client, name, deadline): + name = urllib.parse.quote(name, safe="") + path = f"/apis/batch/v1/namespaces/{client.namespace}/jobs/{name}" + return job_complete(client.get(path, deadline)) + + def check_fleet(client, targets, minimum, deadline): for target in targets: deployment = client.deployment(target["deployment"], deadline) # The chart's Services select app=. Include stray Pods # with that label too; they can receive traffic regardless of ownership. pods = client.pods("app=" + target["deployment"], deadline) - reason = check_deployment(deployment, pods, target["container"], minimum) + reason = check_deployment( + deployment, pods, target, minimum, + ) if reason: return reason return None -def wait_for_fleet(client, targets, minimum, timeout, interval, +def wait_for_fleet(client, configmap, minimum, timeout, interval, clock=time.monotonic, sleep=time.sleep, log=print): # All roles and API requests share one budget, unaffected by wall-clock changes. deadline = clock() + timeout successes = 0 last_reason = "no observation yet" + expected = None while clock() < deadline: try: - reason = check_fleet(client, targets, minimum, deadline) + ordering = load_ordering_config(client, configmap, deadline) + if ordering != expected: + successes = 0 + expected = ordering + if migration_complete(client, ordering["job"], deadline): + reason = check_fleet(client, ordering["targets"], minimum, deadline) + else: + reason = f"migration Job {ordering['job']} has not completed" except urllib.error.HTTPError as error: - # RBAC and Deployments may be applied after the API Pod is created. + # RBAC and Deployments may be applied after the application Pod is created. path = urllib.parse.urlsplit(error.url).path reason = f"Kubernetes API returned HTTP {error.code} for {path}" error.close() @@ -210,9 +321,9 @@ def wait_for_fleet(client, targets, minimum, timeout, interval, if reason is None: successes += 1 if successes == 2: - log("Brainstore startup gate complete: compatible capacity is available", flush=True) + log("Brainstore startup gate complete: compatible rollouts are complete", flush=True) return - last_reason = "confirming compatible capacity (1/2)" + last_reason = "confirming completed compatible rollouts (1/2)" else: # An unhealthy role or failed API read resets the confirmation count. successes = 0 @@ -224,19 +335,19 @@ def wait_for_fleet(client, targets, minimum, timeout, interval, def main(): minimum = release_version(os.environ["BRAINSTORE_MINIMUM_VERSION"]) - targets = json.loads(os.environ["BRAINSTORE_DEPLOYMENTS"]) timeout = int(os.environ["BRAINSTORE_GATE_TIMEOUT_SECONDS"]) interval = int(os.environ["BRAINSTORE_GATE_POLL_SECONDS"]) - if not targets or timeout <= 0 or interval <= 0 or interval >= timeout: + configmap = os.environ["ORDERING_CONFIGMAP"] + if not configmap or timeout <= 0 or interval <= 0 or interval >= timeout: raise ValueError("invalid startup gate configuration") client = KubernetesClient(os.environ["POD_NAMESPACE"], "/var/run/brainstore-gate") - wait_for_fleet(client, targets, minimum, timeout, interval) + wait_for_fleet(client, configmap, minimum, timeout, interval) if __name__ == "__main__": try: main() - except (OSError, ValueError, KeyError, TypeError) as error: + except (OSError, ValueError, KeyError, TypeError, RuntimeError) as error: print(str(error), file=sys.stderr, flush=True) - # Fail closed: Kubernetes retries this init container; the API stays stopped. + # Fail closed: Kubernetes retries this init container; the application stays stopped. sys.exit(1) diff --git a/braintrust/files/job-gate.py b/braintrust/files/job-gate.py new file mode 100644 index 0000000..6438e8e --- /dev/null +++ b/braintrust/files/job-gate.py @@ -0,0 +1,52 @@ +"""Wait for the current content-addressed migration Job; fail closed.""" + +import os +import sys +import time +import urllib.error + +from brainstore_gate import KubernetesClient, load_ordering_config, migration_complete + + +def wait_for_job(client, configmap, timeout, interval, + clock=time.monotonic, sleep=time.sleep, log=print): + deadline = clock() + timeout + reason = "no observation yet" + while clock() < deadline: + stage = "ordering configuration" + try: + ordering = load_ordering_config(client, configmap, deadline) + name = ordering["job"] + stage = f"Job {name}" + complete = migration_complete(client, name, deadline) + if complete and clock() < deadline: + log(f"Job {name} completed successfully", flush=True) + return + reason = f"Job {name} has not completed" + except urllib.error.HTTPError as error: + # Job and RBAC may be applied after the Deployment, including 403/404. + reason = f"{stage}: Kubernetes API returned HTTP {error.code}" + error.close() + except (OSError, ValueError, KeyError, TypeError) as error: + reason = f"{stage}: Kubernetes observation failed: {type(error).__name__}" + log(f"Waiting for migration Job: {reason}", flush=True) + sleep(min(interval, max(0, deadline - clock()))) + raise TimeoutError(f"Migration Job wait timed out after {timeout}s: {reason}") + + +def main(): + timeout = int(os.environ["JOB_TIMEOUT_SECONDS"]) + interval = int(os.environ["JOB_POLL_SECONDS"]) + configmap = os.environ["ORDERING_CONFIGMAP"] + if not configmap or interval <= 0 or timeout <= interval: + raise ValueError("invalid Job gate configuration") + client = KubernetesClient(os.environ["POD_NAMESPACE"], "/var/run/migration-gate") + wait_for_job(client, configmap, timeout, interval) + + +if __name__ == "__main__": + try: + main() + except (OSError, ValueError, KeyError, TypeError, RuntimeError) as error: + print(str(error), file=sys.stderr, flush=True) + sys.exit(1) diff --git a/braintrust/templates/_api-deployment.tpl b/braintrust/templates/_api-deployment.tpl index 7dfc281..2b25ca0 100644 --- a/braintrust/templates/_api-deployment.tpl +++ b/braintrust/templates/_api-deployment.tpl @@ -3,8 +3,11 @@ {{- $root := .root -}} {{- $api := .api -}} {{- $role := .role -}} -{{- $startupGate := include "braintrust.brainstoreStartupGate.config" $root | fromYaml -}} -{{- if $startupGate.enabled -}} +{{- $ordering := include "braintrust.deploymentOrdering.config" $root | fromYaml -}} +{{- if .migrationJob -}} +{{- $_ := set $ordering "enabled" false -}} +{{- end -}} +{{- if $ordering.enabled -}} {{- include "braintrust.brainstoreStartupGate.validateApi" (dict "api" $api) -}} {{- end -}} {{- $customCA := $api.customCA -}} @@ -44,8 +47,8 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - {{- if $startupGate.enabled }} - progressDeadlineSeconds: {{ add (int $startupGate.timeoutSeconds) 300 }} + {{- if $ordering.enabled }} + progressDeadlineSeconds: {{ add 300 (mul 2 (int $ordering.timeoutSeconds)) }} {{- end }} {{- if not (dig "autoscaling" "enabled" false $api) }} replicas: {{ $api.replicas }} @@ -67,7 +70,7 @@ spec: {{- end }} annotations: checksum/config: {{ include (print $root.Template.BasePath "/api-configmap.yaml") $root | sha256sum }} - {{- if $startupGate.enabled }} + {{- if $ordering.enabled }} checksum/brainstore-startup-gate: {{ $root.Files.Get "files/brainstore-startup-gate.py" | sha256sum }} {{- end }} {{- if and (eq $root.Values.cloud "google") $api.enableGcsAuth }} @@ -78,8 +81,8 @@ spec: {{- end }} spec: serviceAccountName: {{ $api.serviceAccount.name }} - {{- if $startupGate.enabled }} - # Only the init container mounts the projected Kubernetes API credential. + {{- if $ordering.enabled }} + # Only the init containers mount the projected Kubernetes API credential. automountServiceAccountToken: false {{- end }} {{- with $api.podSecurityContext }} @@ -107,44 +110,10 @@ spec: affinity: {{- toYaml . | nindent 8 }} {{- end }} - {{- if $startupGate.enabled }} + {{- if $ordering.enabled }} initContainers: - - name: wait-for-brainstore - image: {{ $startupGate.image | quote }} - imagePullPolicy: IfNotPresent - command: ["python3", "-B", "-u", "/opt/brainstore-gate/gate.py"] - securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - runAsNonRoot: true - runAsUser: 65532 - runAsGroup: 65532 - capabilities: - drop: ["ALL"] - seccompProfile: - type: RuntimeDefault - resources: - {{- toYaml $startupGate.resources | nindent 12 }} - env: - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: BRAINSTORE_MINIMUM_VERSION - value: {{ include "braintrust.brainstoreStartupGate.minimumVersion" (dict "gate" $startupGate "api" $api) | quote }} - - name: BRAINSTORE_DEPLOYMENTS - value: {{ include "braintrust.brainstoreStartupGate.targets" $root | quote }} - - name: BRAINSTORE_GATE_TIMEOUT_SECONDS - value: {{ $startupGate.timeoutSeconds | quote }} - - name: BRAINSTORE_GATE_POLL_SECONDS - value: {{ $startupGate.pollIntervalSeconds | quote }} - volumeMounts: - - name: brainstore-gate-code - mountPath: /opt/brainstore-gate - readOnly: true - - name: brainstore-gate-credentials - mountPath: /var/run/brainstore-gate - readOnly: true + {{- include "braintrust.migrations.initContainer" $root | nindent 8 }} + {{- include "braintrust.brainstoreStartupGate.initContainer" (dict "root" $root "api" $api) | nindent 8 }} {{- end }} containers: - name: api @@ -260,22 +229,9 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} volumes: - {{- if or $startupGate.enabled $api.tmpVolume.enabled (and (eq $root.Values.cloud "azure") $root.Values.azure.enableAzureKeyVaultDriver) $customCA.enabled $api.extraVolumes }} - {{- if $startupGate.enabled }} - - name: brainstore-gate-code - configMap: - name: {{ include "braintrust.brainstoreStartupGate.name" $root }} - - name: brainstore-gate-credentials - projected: - sources: - - serviceAccountToken: - path: token - expirationSeconds: 3600 - - configMap: - name: kube-root-ca.crt - items: - - key: ca.crt - path: ca.crt + {{- if or $ordering.enabled $api.tmpVolume.enabled (and (eq $root.Values.cloud "azure") $root.Values.azure.enableAzureKeyVaultDriver) $customCA.enabled $api.extraVolumes }} + {{- if $ordering.enabled }} + {{- include "braintrust.migrations.volumes" $root | nindent 8 }} {{- end }} {{- if $api.tmpVolume.enabled }} - name: tmp-volume diff --git a/braintrust/templates/_brainstore-startup-gate.tpl b/braintrust/templates/_brainstore-startup-gate.tpl index 5f2543c..1325dcd 100644 --- a/braintrust/templates/_brainstore-startup-gate.tpl +++ b/braintrust/templates/_brainstore-startup-gate.tpl @@ -1,39 +1,27 @@ -{{/* Shared configuration. Keep defaults for helm upgrade --reuse-values. */}} -{{- define "braintrust.brainstoreStartupGate.config" -}} -{{- $defaults := dict - "enabled" true - "minimumVersion" "" - "image" "python:3.13-alpine@sha256:79e7a9b9ff1cbceff819f856fb374477792a5967759d94df266de7b7b4120e6f" - "timeoutSeconds" 1200 - "pollIntervalSeconds" 15 - "resources" (dict "requests" (dict "cpu" "25m" "memory" "32Mi") "limits" (dict "cpu" "100m" "memory" "128Mi")) --}} -{{- $gate := mergeOverwrite (deepCopy $defaults) (deepCopy (.Values.api.brainstoreStartupGate | default dict)) -}} -{{- if $gate.enabled -}} -{{- $_ := required "api.brainstoreStartupGate.image is required" $gate.image -}} -{{- range $field := list "timeoutSeconds" "pollIntervalSeconds" -}} -{{- if not (regexMatch "^[1-9][0-9]*$" (toString (index $gate $field))) -}} -{{- fail "api.brainstoreStartupGate timing values must be positive integers" -}} -{{- end -}} -{{- end -}} -{{- if ge (int $gate.pollIntervalSeconds) (int $gate.timeoutSeconds) -}} -{{- fail "api.brainstoreStartupGate timeoutSeconds must exceed pollIntervalSeconds for two observations" -}} -{{- end -}} -{{- end -}} -{{- toYaml $gate -}} -{{- end -}} - {{/* The floor belongs to the API, not the Brainstore image being requested. An API-only image bump must not silently accept the still-old Brainstore fleet. */}} {{- define "braintrust.brainstoreStartupGate.minimumVersion" -}} -{{- $minimum := .gate.minimumVersion | default (first (splitList "@" .api.image.tag)) -}} +{{- $minimum := .gate.minimumVersion | default (first (splitList "@" (toString .api.image.tag))) -}} {{- if not (regexMatch "^v?(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)$" $minimum) -}} -{{- fail "api.brainstoreStartupGate requires a stable API release tag or an explicit stable minimumVersion" -}} +{{- fail "deploymentOrdering requires a stable API release tag or an explicit stable minimumVersion" -}} {{- end -}} {{- trimPrefix "v" $minimum -}} {{- end -}} +{{/* Opaque builds need an operator-supplied release assertion, not a version guess. */}} +{{- define "braintrust.brainstoreStartupGate.brainstoreVersion" -}} +{{- $version := first (splitList "@" (toString .Values.brainstore.image.tag)) -}} +{{- if not (regexMatch "^v?(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)$" $version) -}} +{{- $config := include "braintrust.deploymentOrdering.config" . | fromYaml -}} +{{- $version = required "deploymentOrdering.minimumVersion is required for opaque Brainstore image tags" $config.minimumVersion -}} +{{- end -}} +{{- if not (regexMatch "^v?(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)$" $version) -}} +{{- fail "deploymentOrdering.minimumVersion must be a stable release version" -}} +{{- end -}} +{{- trimPrefix "v" $version -}} +{{- end -}} + {{- define "braintrust.brainstoreStartupGate.name" -}} {{- printf "%s-brainstore-startup" .Release.Name | trunc 63 | trimSuffix "-" -}} {{- end -}} @@ -44,15 +32,82 @@ An API-only image bump must not silently accept the still-old Brainstore fleet. {{- $config := index $.Values.brainstore $role -}} {{- $targets = append $targets (dict "deployment" $config.name "container" (printf "brainstore-%s" $role)) -}} {{- end -}} +{{- if .Values.brainstore.automationwriter -}} +{{- $config := include "braintrust.automationWriter.config" . | fromYaml -}} +{{- $targets = append $targets (dict "deployment" $config.name "container" "brainstore-automationwriter") -}} +{{- end -}} +{{- toJson $targets -}} +{{- end -}} + +{{/* Keep desired identities outside application pod templates: dependency-only +changes must not restart otherwise unchanged applications. */}} +{{- define "braintrust.brainstoreStartupGate.expectedTargets" -}} +{{- $targets := list -}} +{{- range $role := list "reader" "writer" "fastreader" "automationwriter" -}} +{{- if or (ne $role "automationwriter") $.Values.brainstore.automationwriter -}} +{{- $deployment := include (printf "%s/brainstore-%s-deployment.yaml" $.Template.BasePath $role) $ | fromYaml -}} +{{- $replicas := 1 -}} +{{- if not (kindIs "invalid" $deployment.spec.replicas) -}} +{{- $replicas = int $deployment.spec.replicas -}} +{{- end -}} +{{- $targets = append $targets (dict + "deployment" $deployment.metadata.name + "container" (printf "brainstore-%s" $role) + "template" (index $deployment.spec.template.metadata.annotations "braintrust.dev/pod-template") + "replicas" $replicas + "minReadySeconds" (int $deployment.spec.minReadySeconds) +) -}} +{{- end -}} +{{- end -}} {{- toJson $targets -}} {{- end -}} {{- define "braintrust.brainstoreStartupGate.validateApi" -}} {{- $rolling := .api.strategy.rollingUpdate | default dict -}} {{- if or (ne .api.strategy.type "RollingUpdate") (not (has (toString $rolling.maxUnavailable) (list "0" "0%"))) -}} -{{- fail "api.brainstoreStartupGate requires RollingUpdate with maxUnavailable: 0 for every API pool" -}} +{{- fail "deploymentOrdering requires RollingUpdate with maxUnavailable: 0 for every API pool" -}} {{- end -}} {{- if has (toString $rolling.maxSurge) (list "0" "0%" "") -}} -{{- fail "api.brainstoreStartupGate requires nonzero maxSurge for every API pool" -}} +{{- fail "deploymentOrdering requires nonzero maxSurge for every API pool" -}} {{- end -}} {{- end -}} + +{{- define "braintrust.brainstoreStartupGate.initContainer" -}} +{{- $config := include "braintrust.deploymentOrdering.config" .root | fromYaml -}} +- name: wait-for-brainstore + image: {{ $config.image | quote }} + imagePullPolicy: IfNotPresent + command: ["python3", "-B", "-u", "/opt/migration-gate/brainstore_gate.py"] + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault + resources: + {{- toYaml $config.resources | nindent 4 }} + env: + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: BRAINSTORE_MINIMUM_VERSION + value: {{ include "braintrust.brainstoreStartupGate.minimumVersion" (dict "gate" $config "api" .api) | quote }} + - name: ORDERING_CONFIGMAP + value: {{ include "braintrust.deploymentOrdering.name" .root | quote }} + - name: BRAINSTORE_GATE_TIMEOUT_SECONDS + value: {{ $config.timeoutSeconds | quote }} + - name: BRAINSTORE_GATE_POLL_SECONDS + value: {{ $config.pollIntervalSeconds | quote }} + volumeMounts: + - name: migration-gate-code + mountPath: /opt/migration-gate + readOnly: true + - name: migration-gate-credentials + mountPath: /var/run/brainstore-gate + readOnly: true +{{- end -}} diff --git a/braintrust/templates/_deployment-ordering.tpl b/braintrust/templates/_deployment-ordering.tpl new file mode 100644 index 0000000..1320dff --- /dev/null +++ b/braintrust/templates/_deployment-ordering.tpl @@ -0,0 +1,204 @@ +{{/* Defaults also cover pre-feature helm upgrade --reuse-values. */}} +{{- define "braintrust.deploymentOrdering.config" -}} +{{- $defaults := dict + "enabled" true + "minimumVersion" "" + "migrationImage" "" + "timeoutSeconds" 1200 + "pollIntervalSeconds" 15 + "backoffLimit" 2 + "image" "python:3.13-alpine@sha256:79e7a9b9ff1cbceff819f856fb374477792a5967759d94df266de7b7b4120e6f" + "resources" (dict "requests" (dict "cpu" "25m" "memory" "32Mi") "limits" (dict "cpu" "100m" "memory" "128Mi")) +-}} +{{- $config := mergeOverwrite (deepCopy $defaults) (deepCopy (.Values.deploymentOrdering | default dict)) -}} +{{- if $config.enabled -}} +{{- $_ := required "deploymentOrdering.image is required" $config.image -}} +{{- range $field := list "timeoutSeconds" "pollIntervalSeconds" -}} +{{- if not (regexMatch "^[1-9][0-9]*$" (toString (index $config $field))) -}} +{{- fail "deploymentOrdering timing values must be positive integers" -}} +{{- end -}} +{{- end -}} +{{- if ge (int $config.pollIntervalSeconds) (int $config.timeoutSeconds) -}} +{{- fail "deploymentOrdering.timeoutSeconds must exceed pollIntervalSeconds" -}} +{{- end -}} +{{- if not (regexMatch "^(0|[1-9][0-9]*)$" (toString $config.backoffLimit)) -}} +{{- fail "deploymentOrdering.backoffLimit must be a nonnegative integer" -}} +{{- end -}} +{{- end -}} +{{- toYaml $config -}} +{{- end -}} + +{{/* Workloads reference only this stable desired-state ConfigMap, never a Job. */}} +{{- define "braintrust.deploymentOrdering.name" -}} +{{- printf "%s-%s-ordering" (.Release.Name | trunc 40 | trimSuffix "-") (.Release.Name | sha256sum | trunc 8) -}} +{{- end -}} + +{{/* One execution contract supplies both the content hash and the actual Job. +Use a fixed name when hashing to avoid self-referential labels and snapshot refs. */}} +{{- define "braintrust.migrations.execution" -}} +{{- $root := .root -}} +{{- $name := .name -}} +{{- $config := include "braintrust.deploymentOrdering.config" $root | fromYaml -}} +{{- $apiConfig := include (print $root.Template.BasePath "/api-configmap.yaml") $root | fromYaml -}} +{{- $api := include "braintrust.apiPoolConfig" (dict "root" $root "overrides" (dict)) | fromYaml -}} +{{- $deployment := include "braintrust.apiDeployment" (dict "root" $root "api" $api "role" "migration" "migrationJob" true) | fromYaml -}} +{{- $pod := $deployment.spec.template -}} +{{- $container := first $pod.spec.containers -}} +{{- range $key := list "ports" "livenessProbe" "readinessProbe" "startupProbe" -}} +{{- $_ := unset $container $key -}} +{{- end -}} +{{- $_ := set $container "name" "migrate" -}} +{{- $_ := set $container "image" (include "braintrust.migrations.image" $root) -}} +{{- $_ := set $container "command" (list "/braintrust/venv/bin/python" "/braintrust/api-schema/lambda_function.py" "--execute") -}} +{{- $_ := set $container "envFrom" (list (dict "configMapRef" (dict "name" (printf "%s-config" $name)))) -}} +{{- $_ := set $pod.spec "containers" (list $container) -}} +{{- $_ := set $pod.spec "restartPolicy" "Never" -}} +{{- $_ := set $pod.spec "automountServiceAccountToken" false -}} +{{- $_ := unset $pod.spec "topologySpreadConstraints" -}} +{{/* The snapshot's data, not mutable API ConfigMap metadata, drives execution. */}} +{{- $_ := set $pod.metadata.annotations "checksum/config" ($apiConfig.data | toJson | sha256sum) -}} +{{/* A migration pod must never match API Service / Deployment selectors. */}} +{{- $_ := set $pod.metadata.labels "app" $name -}} +{{- $_ := set $pod.metadata.labels "braintrust.dev/api-pool" "migration" -}} +{{- dict "pod" $pod "apiConfig" $apiConfig.data "settings" (dict "backoffLimit" (int $config.backoffLimit) "activeDeadlineSeconds" (int $config.timeoutSeconds)) | toYaml -}} +{{- end -}} + +{{/* Reserve space for the immutable snapshot suffix; hash full release identity +with execution inputs so truncated prefixes cannot collide across releases. */}} +{{- define "braintrust.migrations.name" -}} +{{- $execution := include "braintrust.migrations.execution" (dict "root" . "name" "migration-content") | fromYaml -}} +{{- $identity := dict "release" .Release.Name "namespace" .Release.Namespace "execution" $execution -}} +{{- printf "%s-migrate-%s" (.Release.Name | trunc 31 | trimSuffix "-") ($identity | toJson | sha256sum | trunc 16) -}} +{{- end -}} + +{{/* Migration scripts ship in standalone-api, never in the Brainstore or gate +helper image. Preserve API pins, but translate a newer Brainstore release to +the shared API repository without carrying over the Brainstore digest. */}} +{{- define "braintrust.migrations.image" -}} +{{- $config := include "braintrust.deploymentOrdering.config" . | fromYaml -}} +{{- if $config.enabled -}} +{{- $stableRelease := "^v?(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)$" -}} +{{- $images := list -}} +{{- range $pool := include "braintrust.apiPools" . | fromYamlArray -}} +{{- $images = append $images (dict "repository" $pool.config.image.repository "tag" (toString $pool.config.image.tag) "brainstore" false) -}} +{{- end -}} +{{- $images = append $images (dict "repository" .Values.api.image.repository "tag" (toString .Values.brainstore.image.tag) "brainstore" true) -}} +{{- $selectedImage := "" -}} +{{- $selectedVersion := "" -}} +{{- $major := "" -}} +{{- $opaque := false -}} +{{- range $image := $images -}} +{{- $tag := first (splitList "@" $image.tag) -}} +{{- if regexMatch $stableRelease $tag -}} +{{- $version := trimPrefix "v" $tag -}} +{{- $imageMajor := first (splitList "." $version) -}} +{{- if and $major (ne $major $imageMajor) -}} +{{- fail "deploymentOrdering cannot migrate API and Brainstore service images from different major releases" -}} +{{- end -}} +{{- $major = $imageMajor -}} +{{- if or (not $selectedVersion) (semverCompare (printf ">%s" $selectedVersion) $version) -}} +{{- $selectedVersion = $version -}} +{{- $selectedTag := $image.tag -}} +{{- if $image.brainstore -}} +{{- $selectedTag = $tag -}} +{{- end -}} +{{- $selectedImage = printf "%s:%s" $image.repository $selectedTag -}} +{{- end -}} +{{- else -}} +{{- $opaque = true -}} +{{- end -}} +{{- end -}} +{{- $override := $config.migrationImage -}} +{{- if kindIs "invalid" $override -}} +{{- $override = "" -}} +{{- end -}} +{{- if not (kindIs "string" $override) -}} +{{- fail "deploymentOrdering.migrationImage must be a full standalone-API image reference" -}} +{{- end -}} +{{- if and $opaque (not $override) -}} +{{- fail "deploymentOrdering.migrationImage is required for opaque API or Brainstore service image tags; minimumVersion cannot select a migration bundle" -}} +{{- end -}} +{{- $minimum := $config.minimumVersion | default "" | toString -}} +{{- if and $opaque (not $minimum) -}} +{{- fail "deploymentOrdering.minimumVersion is required for opaque API or Brainstore service image tags" -}} +{{- end -}} +{{- if and $minimum (not (regexMatch $stableRelease $minimum)) -}} +{{- fail "deploymentOrdering.minimumVersion must be a stable release version" -}} +{{- end -}} +{{- if $override -}} +{{- $reference := first (splitList "@" $override) -}} +{{- $lastComponent := last (splitList "/" $reference) -}} +{{- $overrideTag := "" -}} +{{- if contains ":" $lastComponent -}} +{{- $overrideTag = last (splitList ":" $lastComponent) -}} +{{- end -}} +{{- if regexMatch $stableRelease $overrideTag -}} +{{- $requiredVersion := $selectedVersion -}} +{{- if and $minimum (or (not $requiredVersion) (semverCompare (printf ">%s" $requiredVersion) $minimum)) -}} +{{- $requiredVersion = trimPrefix "v" $minimum -}} +{{- end -}} +{{- if and $requiredVersion (semverCompare (printf "<%s" $requiredVersion) $overrideTag) -}} +{{- fail (printf "deploymentOrdering.migrationImage release %s is older than required release %s" $overrideTag $requiredVersion) -}} +{{- end -}} +{{- end -}} +{{- $override -}} +{{- else -}} +{{- $selectedImage -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{- define "braintrust.migrations.initContainer" -}} +{{- $config := include "braintrust.deploymentOrdering.config" . | fromYaml -}} +- name: wait-for-migrations + image: {{ $config.image | quote }} + imagePullPolicy: IfNotPresent + command: ["python3", "-B", "-u", "/opt/migration-gate/gate.py"] + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault + resources: + {{- toYaml $config.resources | nindent 4 }} + env: + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: ORDERING_CONFIGMAP + value: {{ include "braintrust.deploymentOrdering.name" . | quote }} + - name: JOB_TIMEOUT_SECONDS + value: {{ $config.timeoutSeconds | quote }} + - name: JOB_POLL_SECONDS + value: {{ $config.pollIntervalSeconds | quote }} + volumeMounts: + - name: migration-gate-code + mountPath: /opt/migration-gate + readOnly: true + - name: migration-gate-credentials + mountPath: /var/run/migration-gate + readOnly: true +{{- end -}} + +{{- define "braintrust.migrations.volumes" -}} +- name: migration-gate-code + configMap: + name: {{ include "braintrust.deploymentOrdering.name" . }} +- name: migration-gate-credentials + projected: + sources: + - serviceAccountToken: + path: token + expirationSeconds: 3600 + - configMap: + name: kube-root-ca.crt + items: + - key: ca.crt + path: ca.crt +{{- end -}} diff --git a/braintrust/templates/_helpers.tpl b/braintrust/templates/_helpers.tpl index 69d87b8..6b35d51 100644 --- a/braintrust/templates/_helpers.tpl +++ b/braintrust/templates/_helpers.tpl @@ -146,6 +146,11 @@ Internal cluster URL for the AI Gateway service. http://{{ .Values.aiGateway.service.name | default .Values.aiGateway.name }}.{{ include "braintrust.namespace" . }}:{{ .Values.aiGateway.service.port }} {{- end -}} +{{/* Also supports reused values that predate the Gateway ServiceAccount. */}} +{{- define "braintrust.aiGateway.serviceAccountName" -}} +{{- (.Values.aiGateway.serviceAccount | default dict).name | default "braintrust-ai-gateway" -}} +{{- end -}} + {{/* Whether Loop Runtime is enabled. Values reused from a chart release that predates Loop Runtime have no loopRuntime map, which means disabled. @@ -289,6 +294,27 @@ and normal writable-layer/log overhead. {{- toYaml $resources -}} {{- end -}} +{{/* Merge pod annotations; content identity rejects stale Ready rollouts. */}} +{{- define "braintrust.brainstore.podAnnotations" -}} +{{- $root := .root -}} +{{- $annotations := dict "checksum/config" (include (printf "%s/brainstore-%s-configmap.yaml" $root.Template.BasePath .role) $root | sha256sum) -}} +{{- if eq $root.Values.cloud "google" -}} +{{- $_ := set $annotations "iam.gke.io/gcp-service-account" (required "brainstore.serviceAccount.googleServiceAccount is required when cloud is google" $root.Values.brainstore.serviceAccount.googleServiceAccount) -}} +{{- end -}} +{{- $annotations = mergeOverwrite $annotations (deepCopy (.config.annotations.pod | default dict)) -}} +{{- $ordering := include "braintrust.deploymentOrdering.config" $root | fromYaml -}} +{{- if $ordering.enabled -}} +{{- $_ := unset $annotations "braintrust.dev/pod-template" -}} +{{- $_ := set $annotations "braintrust.dev/brainstore-version" (include "braintrust.brainstoreStartupGate.brainstoreVersion" $root) -}} +{{- if not ($root.brainstoreFingerprint | default false) -}} +{{- $context := merge (dict "brainstoreFingerprint" true) $root -}} +{{- $deployment := include (printf "%s/brainstore-%s-deployment.yaml" $root.Template.BasePath .role) $context | fromYaml -}} +{{- $_ := set $annotations "braintrust.dev/pod-template" ($deployment.spec.template | toJson | sha256sum) -}} +{{- end -}} +{{- end }}{{/* ordering.enabled */}} +{{- toYaml $annotations -}} +{{- end -}} + {{/* Defaults for the optional automation writer pool. diff --git a/braintrust/templates/ai-gateway-deployment.yaml b/braintrust/templates/ai-gateway-deployment.yaml index ef18994..ff5f19d 100644 --- a/braintrust/templates/ai-gateway-deployment.yaml +++ b/braintrust/templates/ai-gateway-deployment.yaml @@ -1,3 +1,4 @@ +{{- $ordering := include "braintrust.deploymentOrdering.config" . | fromYaml -}} {{- if .Values.aiGateway.enabled }} --- apiVersion: apps/v1 @@ -14,6 +15,9 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: + {{- if $ordering.enabled }} + progressDeadlineSeconds: {{ add 600 (mul 2 (int $ordering.timeoutSeconds)) }} + {{- end }} {{- if not (dig "enabled" true (.Values.aiGateway.autoscaling | default dict)) }} replicas: {{ .Values.aiGateway.replicas }} {{- end }} @@ -38,6 +42,10 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} spec: + serviceAccountName: {{ include "braintrust.aiGateway.serviceAccountName" . }} + {{- if $ordering.enabled }} + automountServiceAccountToken: false + {{- end }} {{- with .Values.aiGateway.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} @@ -50,6 +58,11 @@ spec: affinity: {{- toYaml . | nindent 8 }} {{- end }} + {{- if $ordering.enabled }} + initContainers: + {{- include "braintrust.migrations.initContainer" . | nindent 8 }} + {{- include "braintrust.brainstoreStartupGate.initContainer" (dict "root" . "api" .Values.api) | nindent 8 }} + {{- end }} containers: - name: ai-gateway image: "{{ .Values.aiGateway.image.repository }}:{{ .Values.aiGateway.image.tag }}" @@ -92,9 +105,12 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} volumes: - {{- with .Values.aiGateway.extraVolumes }} - {{- toYaml . | nindent 8 }} - {{- else }} + {{- if $ordering.enabled }} + {{- include "braintrust.migrations.volumes" . | nindent 8 }} + {{- end }} + {{- if .Values.aiGateway.extraVolumes }} + {{- toYaml .Values.aiGateway.extraVolumes | nindent 8 }} + {{- else if not $ordering.enabled }} [] {{- end }} {{- end }} diff --git a/braintrust/templates/ai-gateway-serviceaccount.yaml b/braintrust/templates/ai-gateway-serviceaccount.yaml new file mode 100644 index 0000000..068fe22 --- /dev/null +++ b/braintrust/templates/ai-gateway-serviceaccount.yaml @@ -0,0 +1,15 @@ +{{- if .Values.aiGateway.enabled }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "braintrust.aiGateway.serviceAccountName" . }} + namespace: {{ include "braintrust.namespace" . }} + {{- with (mergeOverwrite (deepCopy .Values.global.labels) (deepCopy .Values.aiGateway.labels)) }} + labels: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.aiGateway.annotations.serviceaccount }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/braintrust/templates/brainstore-automationwriter-deployment.yaml b/braintrust/templates/brainstore-automationwriter-deployment.yaml index e8fd45a..4317255 100644 --- a/braintrust/templates/brainstore-automationwriter-deployment.yaml +++ b/braintrust/templates/brainstore-automationwriter-deployment.yaml @@ -1,3 +1,4 @@ +{{- $ordering := include "braintrust.deploymentOrdering.config" . | fromYaml -}} {{/* Omit this Deployment when brainstore.automationwriter is absent. helm upgrade --reuse-values from a chart that predates the pool does not merge the new @@ -23,7 +24,7 @@ metadata: spec: replicas: {{ $aw.replicas }} minReadySeconds: {{ $rollout.minReadySeconds }} - progressDeadlineSeconds: {{ $rollout.progressDeadlineSeconds }} + progressDeadlineSeconds: {{ add (int $rollout.progressDeadlineSeconds) (ternary (int $ordering.timeoutSeconds) 0 $ordering.enabled) }} strategy: type: {{ $rollout.strategy.type }} {{- if eq $rollout.strategy.type "RollingUpdate" }} @@ -47,13 +48,7 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} annotations: - checksum/config: {{ include (print $.Template.BasePath "/brainstore-automationwriter-configmap.yaml") . | sha256sum }} - {{- if eq .Values.cloud "google" }} - iam.gke.io/gcp-service-account: {{ required "brainstore.serviceAccount.googleServiceAccount is required when cloud is google" .Values.brainstore.serviceAccount.googleServiceAccount }} - {{- end }} - {{- with $aw.annotations.pod }} - {{- toYaml . | nindent 8 }} - {{- end }} + {{- include "braintrust.brainstore.podAnnotations" (dict "root" . "role" "automationwriter" "config" $aw) | nindent 8 }} spec: serviceAccountName: {{ .Values.brainstore.serviceAccount.name }} {{- with $aw.podSecurityContext }} @@ -79,6 +74,10 @@ spec: affinity: {{- toYaml . | nindent 8 }} {{- end }} + {{- if $ordering.enabled }} + initContainers: + {{- include "braintrust.migrations.initContainer" . | nindent 8 }} + {{- end }} containers: - name: brainstore-automationwriter image: "{{ .Values.brainstore.image.repository }}:{{ .Values.brainstore.image.tag }}" @@ -176,6 +175,9 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} volumes: + {{- if $ordering.enabled }} + {{- include "braintrust.migrations.volumes" . | nindent 8 }} + {{- end }} - name: cache-volume {{- /* Azure local disk is required only once this pool has replicas. A disabled pool must render without a size. */}} {{- if and (eq .Values.cloud "azure") .Values.azure.enableAzureContainerStorageDriver (gt (int ($aw.replicas | default 0)) 0) }} diff --git a/braintrust/templates/brainstore-fastreader-deployment.yaml b/braintrust/templates/brainstore-fastreader-deployment.yaml index 6ca1d5e..0049fb5 100644 --- a/braintrust/templates/brainstore-fastreader-deployment.yaml +++ b/braintrust/templates/brainstore-fastreader-deployment.yaml @@ -1,3 +1,4 @@ +{{- $ordering := include "braintrust.deploymentOrdering.config" . | fromYaml -}} {{- $rollout := include "braintrust.deploymentRollout.config" (dict "config" .Values.brainstore.fastreader) | fromYaml -}} {{- include "braintrust.deploymentRollout.validate" (dict "path" "brainstore.fastreader" "config" $rollout) }} apiVersion: apps/v1 @@ -16,7 +17,7 @@ metadata: spec: replicas: {{ .Values.brainstore.fastreader.replicas }} minReadySeconds: {{ $rollout.minReadySeconds }} - progressDeadlineSeconds: {{ $rollout.progressDeadlineSeconds }} + progressDeadlineSeconds: {{ add (int $rollout.progressDeadlineSeconds) (ternary (int $ordering.timeoutSeconds) 0 $ordering.enabled) }} strategy: type: {{ $rollout.strategy.type }} {{- if eq $rollout.strategy.type "RollingUpdate" }} @@ -40,13 +41,7 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} annotations: - checksum/config: {{ include (print $.Template.BasePath "/brainstore-fastreader-configmap.yaml") . | sha256sum }} - {{- if eq .Values.cloud "google" }} - iam.gke.io/gcp-service-account: {{ required "brainstore.serviceAccount.googleServiceAccount is required when cloud is google" .Values.brainstore.serviceAccount.googleServiceAccount }} - {{- end }} - {{- with .Values.brainstore.fastreader.annotations.pod }} - {{- toYaml . | nindent 8 }} - {{- end }} + {{- include "braintrust.brainstore.podAnnotations" (dict "root" . "role" "fastreader" "config" .Values.brainstore.fastreader) | nindent 8 }} spec: serviceAccountName: {{ .Values.brainstore.serviceAccount.name }} {{- with .Values.brainstore.fastreader.podSecurityContext }} @@ -72,6 +67,10 @@ spec: affinity: {{- toYaml . | nindent 8 }} {{- end }} + {{- if $ordering.enabled }} + initContainers: + {{- include "braintrust.migrations.initContainer" . | nindent 8 }} + {{- end }} containers: - name: brainstore-fastreader image: "{{ .Values.brainstore.image.repository }}:{{ .Values.brainstore.image.tag }}" @@ -161,6 +160,9 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} volumes: + {{- if $ordering.enabled }} + {{- include "braintrust.migrations.volumes" . | nindent 8 }} + {{- end }} - name: cache-volume {{- if and (eq .Values.cloud "azure") .Values.azure.enableAzureContainerStorageDriver }} ephemeral: diff --git a/braintrust/templates/brainstore-reader-deployment.yaml b/braintrust/templates/brainstore-reader-deployment.yaml index f0342a7..2b984a3 100644 --- a/braintrust/templates/brainstore-reader-deployment.yaml +++ b/braintrust/templates/brainstore-reader-deployment.yaml @@ -1,3 +1,4 @@ +{{- $ordering := include "braintrust.deploymentOrdering.config" . | fromYaml -}} {{- $rollout := include "braintrust.deploymentRollout.config" (dict "config" .Values.brainstore.reader) | fromYaml -}} {{- include "braintrust.deploymentRollout.validate" (dict "path" "brainstore.reader" "config" $rollout) }} apiVersion: apps/v1 @@ -16,7 +17,7 @@ metadata: spec: replicas: {{ .Values.brainstore.reader.replicas }} minReadySeconds: {{ $rollout.minReadySeconds }} - progressDeadlineSeconds: {{ $rollout.progressDeadlineSeconds }} + progressDeadlineSeconds: {{ add (int $rollout.progressDeadlineSeconds) (ternary (int $ordering.timeoutSeconds) 0 $ordering.enabled) }} strategy: type: {{ $rollout.strategy.type }} {{- if eq $rollout.strategy.type "RollingUpdate" }} @@ -40,13 +41,7 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} annotations: - checksum/config: {{ include (print $.Template.BasePath "/brainstore-reader-configmap.yaml") . | sha256sum }} - {{- if eq .Values.cloud "google" }} - iam.gke.io/gcp-service-account: {{ required "brainstore.serviceAccount.googleServiceAccount is required when cloud is google" .Values.brainstore.serviceAccount.googleServiceAccount }} - {{- end }} - {{- with .Values.brainstore.reader.annotations.pod }} - {{- toYaml . | nindent 8 }} - {{- end }} + {{- include "braintrust.brainstore.podAnnotations" (dict "root" . "role" "reader" "config" .Values.brainstore.reader) | nindent 8 }} spec: serviceAccountName: {{ .Values.brainstore.serviceAccount.name }} {{- with .Values.brainstore.reader.podSecurityContext }} @@ -72,6 +67,10 @@ spec: affinity: {{- toYaml . | nindent 8 }} {{- end }} + {{- if $ordering.enabled }} + initContainers: + {{- include "braintrust.migrations.initContainer" . | nindent 8 }} + {{- end }} containers: - name: brainstore-reader image: "{{ .Values.brainstore.image.repository }}:{{ .Values.brainstore.image.tag }}" @@ -161,6 +160,9 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} volumes: + {{- if $ordering.enabled }} + {{- include "braintrust.migrations.volumes" . | nindent 8 }} + {{- end }} - name: cache-volume {{- if and (eq .Values.cloud "azure") .Values.azure.enableAzureContainerStorageDriver }} ephemeral: diff --git a/braintrust/templates/brainstore-startup-gate-configmap.yaml b/braintrust/templates/brainstore-startup-gate-configmap.yaml index bd61aa8..4b761da 100644 --- a/braintrust/templates/brainstore-startup-gate-configmap.yaml +++ b/braintrust/templates/brainstore-startup-gate-configmap.yaml @@ -1,15 +1,25 @@ -{{- $gate := include "braintrust.brainstoreStartupGate.config" . | fromYaml -}} -{{- if $gate.enabled }} +{{- $namespace := include "braintrust.namespace" . -}} +{{- $name := include "braintrust.brainstoreStartupGate.name" . -}} +{{- $existing := lookup "v1" "ConfigMap" $namespace $name -}} +{{- if $existing }} +{{/* Old ReplicaSets require their original script and environment contract. */}} apiVersion: v1 kind: ConfigMap metadata: - name: {{ include "braintrust.brainstoreStartupGate.name" . }} - namespace: {{ include "braintrust.namespace" . }} + name: {{ $name }} + namespace: {{ $namespace }} {{- with (mergeOverwrite (deepCopy .Values.global.labels) (deepCopy .Values.api.labels)) }} labels: {{- toYaml . | nindent 4 }} {{- end }} + annotations: + {{- mergeOverwrite (deepCopy (.Values.api.annotations.configmap | default dict)) (dict "helm.sh/resource-policy" "keep") | toYaml | nindent 4 }} +{{- if hasKey $existing "data" }} data: - gate.py: | - {{- .Files.Get "files/brainstore-startup-gate.py" | nindent 4 }} + {{- toYaml $existing.data | nindent 2 }} +{{- end }} +{{- if hasKey $existing "binaryData" }} +binaryData: + {{- toYaml $existing.binaryData | nindent 2 }} +{{- end }} {{- end }} diff --git a/braintrust/templates/brainstore-startup-gate-rbac.yaml b/braintrust/templates/brainstore-startup-gate-rbac.yaml index b81e8a2..611ac56 100644 --- a/braintrust/templates/brainstore-startup-gate-rbac.yaml +++ b/braintrust/templates/brainstore-startup-gate-rbac.yaml @@ -1,4 +1,4 @@ -{{- $gate := include "braintrust.brainstoreStartupGate.config" . | fromYaml -}} +{{- $gate := include "braintrust.deploymentOrdering.config" . | fromYaml -}} {{- if $gate.enabled }} apiVersion: rbac.authorization.k8s.io/v1 kind: Role @@ -13,9 +13,9 @@ rules: - apiGroups: ["apps"] resources: ["deployments"] resourceNames: - - {{ .Values.brainstore.reader.name | quote }} - - {{ .Values.brainstore.writer.name | quote }} - - {{ .Values.brainstore.fastreader.name | quote }} + {{- range $target := include "braintrust.brainstoreStartupGate.targets" . | fromJsonArray }} + - {{ $target.deployment | quote }} + {{- end }} verbs: ["get"] - apiGroups: [""] resources: ["pods"] @@ -39,6 +39,12 @@ subjects: {{- range $pool := include "braintrust.apiPools" . | fromYamlArray -}} {{- $_ := set $accounts $pool.config.serviceAccount.name true -}} {{- end -}} + {{- if .Values.aiGateway.enabled -}} + {{- $_ := set $accounts (include "braintrust.aiGateway.serviceAccountName" .) true -}} + {{- end -}} + {{- if include "braintrust.loopRuntime.enabled" . -}} + {{- $_ := set $accounts .Values.loopRuntime.serviceAccount.name true -}} + {{- end -}} {{- range $name := keys $accounts | sortAlpha }} - kind: ServiceAccount name: {{ $name }} diff --git a/braintrust/templates/brainstore-writer-deployment.yaml b/braintrust/templates/brainstore-writer-deployment.yaml index f42b692..d3c6abf 100644 --- a/braintrust/templates/brainstore-writer-deployment.yaml +++ b/braintrust/templates/brainstore-writer-deployment.yaml @@ -1,3 +1,4 @@ +{{- $ordering := include "braintrust.deploymentOrdering.config" . | fromYaml -}} {{- $rollout := include "braintrust.deploymentRollout.config" (dict "config" .Values.brainstore.writer) | fromYaml -}} {{- include "braintrust.deploymentRollout.validate" (dict "path" "brainstore.writer" "config" $rollout) }} apiVersion: apps/v1 @@ -16,7 +17,7 @@ metadata: spec: replicas: {{ .Values.brainstore.writer.replicas }} minReadySeconds: {{ $rollout.minReadySeconds }} - progressDeadlineSeconds: {{ $rollout.progressDeadlineSeconds }} + progressDeadlineSeconds: {{ add (int $rollout.progressDeadlineSeconds) (ternary (int $ordering.timeoutSeconds) 0 $ordering.enabled) }} strategy: type: {{ $rollout.strategy.type }} {{- if eq $rollout.strategy.type "RollingUpdate" }} @@ -40,13 +41,7 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} annotations: - checksum/config: {{ include (print $.Template.BasePath "/brainstore-writer-configmap.yaml") . | sha256sum }} - {{- if eq .Values.cloud "google" }} - iam.gke.io/gcp-service-account: {{ required "brainstore.serviceAccount.googleServiceAccount is required when cloud is google" .Values.brainstore.serviceAccount.googleServiceAccount }} - {{- end }} - {{- with .Values.brainstore.writer.annotations.pod }} - {{- toYaml . | nindent 8 }} - {{- end }} + {{- include "braintrust.brainstore.podAnnotations" (dict "root" . "role" "writer" "config" .Values.brainstore.writer) | nindent 8 }} spec: serviceAccountName: {{ .Values.brainstore.serviceAccount.name }} {{- with .Values.brainstore.writer.podSecurityContext }} @@ -72,6 +67,10 @@ spec: affinity: {{- toYaml . | nindent 8 }} {{- end }} + {{- if $ordering.enabled }} + initContainers: + {{- include "braintrust.migrations.initContainer" . | nindent 8 }} + {{- end }} containers: - name: brainstore-writer image: "{{ .Values.brainstore.image.repository }}:{{ .Values.brainstore.image.tag }}" @@ -161,6 +160,9 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} volumes: + {{- if $ordering.enabled }} + {{- include "braintrust.migrations.volumes" . | nindent 8 }} + {{- end }} - name: cache-volume {{- if and (eq .Values.cloud "azure") .Values.azure.enableAzureContainerStorageDriver }} ephemeral: diff --git a/braintrust/templates/loop-runtime-deployment.yaml b/braintrust/templates/loop-runtime-deployment.yaml index e82e4b4..659ad4c 100644 --- a/braintrust/templates/loop-runtime-deployment.yaml +++ b/braintrust/templates/loop-runtime-deployment.yaml @@ -1,3 +1,4 @@ +{{- $ordering := include "braintrust.deploymentOrdering.config" . | fromYaml -}} {{- include "braintrust.loopRuntime.validate" . -}} {{- if include "braintrust.loopRuntime.enabled" . -}} {{- $autoscaling := include "braintrust.loopRuntime.autoscaling" . | fromYaml -}} @@ -16,6 +17,9 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: + {{- if $ordering.enabled }} + progressDeadlineSeconds: {{ add 600 (mul 2 (int $ordering.timeoutSeconds)) }} + {{- end }} {{- if not $autoscaling }} replicas: {{ ternary 1 .Values.loopRuntime.replicas (has .Values.loopRuntime.replicas (list nil "")) }} {{- end }} @@ -40,6 +44,9 @@ spec: {{- end }} spec: serviceAccountName: {{ .Values.loopRuntime.serviceAccount.name }} + {{- if $ordering.enabled }} + automountServiceAccountToken: false + {{- end }} # The runtime drains active turns for up to 90 seconds after SIGTERM. terminationGracePeriodSeconds: 120 securityContext: @@ -61,6 +68,11 @@ spec: affinity: {{- toYaml . | nindent 8 }} {{- end }} + {{- if $ordering.enabled }} + initContainers: + {{- include "braintrust.migrations.initContainer" . | nindent 8 }} + {{- include "braintrust.brainstoreStartupGate.initContainer" (dict "root" . "api" .Values.api) | nindent 8 }} + {{- end }} containers: - name: loop-runtime image: "{{ .Values.loopRuntime.image.repository }}:{{ .Values.loopRuntime.image.tag }}" @@ -148,6 +160,9 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} volumes: + {{- if $ordering.enabled }} + {{- include "braintrust.migrations.volumes" . | nindent 8 }} + {{- end }} - name: state emptyDir: sizeLimit: 2Gi diff --git a/braintrust/templates/migration-gate.yaml b/braintrust/templates/migration-gate.yaml new file mode 100644 index 0000000..8c9cdf4 --- /dev/null +++ b/braintrust/templates/migration-gate.yaml @@ -0,0 +1,63 @@ +{{- $config := include "braintrust.deploymentOrdering.config" . | fromYaml -}} +{{- if $config.enabled -}} +{{- $name := include "braintrust.deploymentOrdering.name" . -}} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ $name }} + namespace: {{ include "braintrust.namespace" . }} + labels: + braintrust.dev/migration-release: {{ .Release.Name | quote }} +data: + ordering.json: {{ dict "job" (include "braintrust.migrations.name" .) "targets" (include "braintrust.brainstoreStartupGate.expectedTargets" . | fromJsonArray) | toJson | quote }} + gate.py: | + {{- .Files.Get "files/job-gate.py" | nindent 4 }} + brainstore_gate.py: | + {{- .Files.Get "files/brainstore-startup-gate.py" | nindent 4 }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ $name }} + namespace: {{ include "braintrust.namespace" . }} + labels: + braintrust.dev/migration-release: {{ .Release.Name | quote }} +rules: + - apiGroups: ["batch"] + resources: ["jobs"] + resourceNames: + - {{ include "braintrust.migrations.name" . | quote }} + verbs: ["get"] + - apiGroups: [""] + resources: ["configmaps"] + resourceNames: [{{ $name | quote }}] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ $name }} + namespace: {{ include "braintrust.namespace" . }} + labels: + braintrust.dev/migration-release: {{ .Release.Name | quote }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ $name }} +subjects: + {{- $accounts := dict .Values.brainstore.serviceAccount.name true -}} + {{- range $pool := include "braintrust.apiPools" . | fromYamlArray -}} + {{- $_ := set $accounts $pool.config.serviceAccount.name true -}} + {{- end -}} + {{- if include "braintrust.loopRuntime.enabled" . -}} + {{- $_ := set $accounts .Values.loopRuntime.serviceAccount.name true -}} + {{- end -}} + {{- if .Values.aiGateway.enabled -}} + {{- $_ := set $accounts (include "braintrust.aiGateway.serviceAccountName" .) true -}} + {{- end -}} + {{- range $account := keys $accounts | sortAlpha }} + - kind: ServiceAccount + name: {{ $account }} + namespace: {{ include "braintrust.namespace" $ }} + {{- end }} +{{- end }} diff --git a/braintrust/templates/migration-job.yaml b/braintrust/templates/migration-job.yaml new file mode 100644 index 0000000..a9bc86c --- /dev/null +++ b/braintrust/templates/migration-job.yaml @@ -0,0 +1,34 @@ +{{- $config := include "braintrust.deploymentOrdering.config" . | fromYaml -}} +{{- if $config.enabled -}} +{{- $name := include "braintrust.migrations.name" . -}} +{{/* Reuse exactly the execution contract used to content-address this Job. */}} +{{- $execution := include "braintrust.migrations.execution" (dict "root" . "name" $name) | fromYaml -}} +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ $name }} + namespace: {{ include "braintrust.namespace" . }} + annotations: + helm.sh/resource-policy: keep + labels: + {{- mergeOverwrite (deepCopy .Values.global.labels) (deepCopy .Values.api.labels) (dict "braintrust.dev/migration-release" .Release.Name) | toYaml | nindent 4 }} +spec: + {{- toYaml $execution.settings | nindent 2 }} + template: + {{- toYaml $execution.pod | nindent 4 }} +--- +{{/* Successful content-addressed Jobs and their configuration snapshots are +reused across upgrades and rollbacks. Secrets remain Secret references. */}} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ $name }}-config + namespace: {{ include "braintrust.namespace" . }} + annotations: + helm.sh/resource-policy: keep + labels: + braintrust.dev/migration-release: {{ .Release.Name | quote }} +immutable: true +data: + {{- toYaml $execution.apiConfig | nindent 2 }} +{{- end }} diff --git a/braintrust/tests/ai-gateway-serviceaccount_test.yaml b/braintrust/tests/ai-gateway-serviceaccount_test.yaml new file mode 100644 index 0000000..5efffcd --- /dev/null +++ b/braintrust/tests/ai-gateway-serviceaccount_test.yaml @@ -0,0 +1,84 @@ +suite: AI Gateway service account lifecycle +templates: + - ai-gateway-serviceaccount.yaml + - ai-gateway-deployment.yaml + - ai-gateway-configmap.yaml + - migration-gate.yaml + - brainstore-startup-gate-rbac.yaml + - api-configmap.yaml + - brainstore-reader-deployment.yaml + - brainstore-reader-configmap.yaml + - brainstore-writer-deployment.yaml + - brainstore-writer-configmap.yaml + - brainstore-fastreader-deployment.yaml + - brainstore-fastreader-configmap.yaml + - brainstore-automationwriter-deployment.yaml + - brainstore-automationwriter-configmap.yaml +tests: + - it: omits Gateway identity and workload when Gateway is disabled + set: + aiGateway.enabled: false + asserts: + - hasDocuments: + count: 0 + template: ai-gateway-serviceaccount.yaml + - hasDocuments: + count: 0 + template: ai-gateway-deployment.yaml + + - it: keeps Gateway identity without granting ordering permissions when ordering is disabled + release: + namespace: install-namespace + set: + aiGateway.enabled: true + aiGateway.serviceAccount.name: gateway-identity + deploymentOrdering.enabled: false + global.createNamespace: true + global.namespace: workload-namespace + asserts: + - equal: + path: metadata.name + value: gateway-identity + template: ai-gateway-serviceaccount.yaml + - equal: + path: metadata.namespace + value: workload-namespace + template: ai-gateway-serviceaccount.yaml + - equal: + path: spec.template.spec.serviceAccountName + value: gateway-identity + template: ai-gateway-deployment.yaml + - isNull: + path: spec.template.spec.initContainers + template: ai-gateway-deployment.yaml + - hasDocuments: + count: 0 + template: migration-gate.yaml + - hasDocuments: + count: 0 + template: brainstore-startup-gate-rbac.yaml + + - it: grants both ordered waits to the workload identity rather than the namespace default + release: + namespace: workload-namespace + set: + aiGateway.enabled: true + aiGateway.serviceAccount.name: gateway-identity + api.serviceAccount.name: api-identity + brainstore.serviceAccount.name: brainstore-identity + asserts: + - equal: + path: subjects + value: + - {kind: ServiceAccount, name: api-identity, namespace: workload-namespace} + - {kind: ServiceAccount, name: brainstore-identity, namespace: workload-namespace} + - {kind: ServiceAccount, name: gateway-identity, namespace: workload-namespace} + template: migration-gate.yaml + documentIndex: 2 + - equal: + path: subjects + value: + - {kind: ServiceAccount, name: api-identity, namespace: workload-namespace} + - {kind: ServiceAccount, name: gateway-identity, namespace: workload-namespace} + template: brainstore-startup-gate-rbac.yaml + documentIndex: 1 diff --git a/braintrust/tests/api-autoscaling_test.yaml b/braintrust/tests/api-autoscaling_test.yaml index 8b7ce62..299d7f7 100644 --- a/braintrust/tests/api-autoscaling_test.yaml +++ b/braintrust/tests/api-autoscaling_test.yaml @@ -175,6 +175,9 @@ tests: containerPort: 8001 - equal: path: spec.template.spec.initContainers[0].name + value: wait-for-migrations + - equal: + path: spec.template.spec.initContainers[1].name value: wait-for-brainstore - contains: path: spec.template.spec.containers[0].env @@ -189,7 +192,7 @@ tests: set: cloud: google api.autoscaling.enabled: true - api.brainstoreStartupGate.enabled: false + deploymentOrdering.enabled: false api.extraEnvVars: - name: ENABLE_PROMETHEUS_METRICS value: "false" @@ -198,6 +201,8 @@ tests: release: namespace: "braintrust" asserts: + - isNull: + path: spec.template.spec.initContainers - equal: path: spec.template.spec.containers[0].env[-1].name value: ENABLE_PROMETHEUS_METRICS @@ -397,6 +402,10 @@ tests: template: api-deployment.yaml - equal: path: spec.template.spec.initContainers[0].name + value: wait-for-migrations + template: api-deployment.yaml + - equal: + path: spec.template.spec.initContainers[1].name value: wait-for-brainstore template: api-deployment.yaml - equal: diff --git a/braintrust/tests/brainstore-automationwriter-routing_test.yaml b/braintrust/tests/brainstore-automationwriter-routing_test.yaml index 44b3d5d..cbdfe87 100644 --- a/braintrust/tests/brainstore-automationwriter-routing_test.yaml +++ b/braintrust/tests/brainstore-automationwriter-routing_test.yaml @@ -86,35 +86,6 @@ tests: path: data.BRAINSTORE_AUTOMATION_WRITER_URL value: http://custom-pool.created-namespace:4000 - - it: should preserve existing API startup targets with automation writers enabled - template: api-deployment.yaml - set: - <<: *enabled - brainstore.automationwriter.name: custom-pool - asserts: &startup-targets - - contains: - path: spec.template.spec.initContainers[0].env - content: - name: BRAINSTORE_DEPLOYMENTS - value: '[{"container":"brainstore-reader","deployment":"brainstore-reader"},{"container":"brainstore-writer","deployment":"brainstore-writer"},{"container":"brainstore-fastreader","deployment":"brainstore-fastreader"}]' - - - it: should preserve background API startup targets with automation writers enabled - template: api-deployment.yaml - documentIndex: 2 - set: - <<: *enabled - api.workloadIsolation.enabled: true - brainstore.automationwriter.name: custom-pool - asserts: *startup-targets - - - it: should preserve ingest API startup targets with automation writers enabled - template: api-deployment.yaml - documentIndex: 1 - set: - <<: *enabled - api.workloadIsolation.enabled: true - brainstore.automationwriter.name: custom-pool - asserts: *startup-targets - it: should preserve gate permissions with a custom automation deployment enabled template: brainstore-startup-gate-rbac.yaml @@ -125,7 +96,7 @@ tests: asserts: - equal: path: rules[0].resourceNames - value: [brainstore-reader, brainstore-writer, brainstore-fastreader] + value: [brainstore-reader, brainstore-writer, brainstore-fastreader, custom-pool] - it: should preserve gate permissions with a partial reused pool enabled template: brainstore-startup-gate-rbac.yaml @@ -135,25 +106,10 @@ tests: asserts: - equal: path: rules[0].resourceNames - value: [brainstore-reader, brainstore-writer, brainstore-fastreader] - - - it: should omit disabled automation writers from gate targets - template: api-deployment.yaml - asserts: *startup-targets + value: [brainstore-reader, brainstore-writer, brainstore-fastreader, brainstore-automationwriter] - - it: should omit a missing reused pool from gate targets - template: api-deployment.yaml - values: - - __fixtures__/pre-automationwriter-values.yaml - asserts: *startup-targets - - - it: should omit null replicas from gate targets - template: api-deployment.yaml - set: - brainstore.automationwriter.replicas: null - asserts: *startup-targets - - it: should omit empty replicas from gate permissions + - it: should track a rendered zero-replica pool until it finishes draining template: brainstore-startup-gate-rbac.yaml documentIndex: 0 set: @@ -161,13 +117,13 @@ tests: asserts: - equal: path: rules[0].resourceNames - value: [brainstore-reader, brainstore-writer, brainstore-fastreader] + value: [brainstore-reader, brainstore-writer, brainstore-fastreader, brainstore-automationwriter] - it: should still allow an explicitly disabled startup gate template: api-deployment.yaml set: <<: *enabled - api.brainstoreStartupGate.enabled: false + deploymentOrdering.enabled: false asserts: - isNull: path: spec.template.spec.initContainers diff --git a/braintrust/tests/brainstore-automationwriter_test.yaml b/braintrust/tests/brainstore-automationwriter_test.yaml index 6f83e00..d09f389 100644 --- a/braintrust/tests/brainstore-automationwriter_test.yaml +++ b/braintrust/tests/brainstore-automationwriter_test.yaml @@ -2,6 +2,9 @@ suite: test Brainstore Automation Writer deployment template templates: - brainstore-automationwriter-deployment.yaml - brainstore-automationwriter-configmap.yaml +# Test workload details independently of release ordering. +set: + deploymentOrdering.enabled: false tests: - it: should render Brainstore Automation Writer deployment with default values template: brainstore-automationwriter-deployment.yaml diff --git a/braintrust/tests/brainstore-fastreader_test.yaml b/braintrust/tests/brainstore-fastreader_test.yaml index 295bb3f..ff915a5 100644 --- a/braintrust/tests/brainstore-fastreader_test.yaml +++ b/braintrust/tests/brainstore-fastreader_test.yaml @@ -2,6 +2,9 @@ suite: test Brainstore Fast Reader deployment template templates: - brainstore-fastreader-deployment.yaml - brainstore-fastreader-configmap.yaml +# Test workload details independently of release ordering. +set: + deploymentOrdering.enabled: false tests: - it: should render Brainstore Fast Reader deployment with default values template: brainstore-fastreader-deployment.yaml diff --git a/braintrust/tests/brainstore-reader_test.yaml b/braintrust/tests/brainstore-reader_test.yaml index 5575af9..c1d1747 100644 --- a/braintrust/tests/brainstore-reader_test.yaml +++ b/braintrust/tests/brainstore-reader_test.yaml @@ -2,6 +2,9 @@ suite: test Brainstore Reader deployment template templates: - brainstore-reader-deployment.yaml - brainstore-reader-configmap.yaml +# Test workload details independently of release ordering. +set: + deploymentOrdering.enabled: false tests: - it: should render Brainstore Reader deployment with default values template: brainstore-reader-deployment.yaml diff --git a/braintrust/tests/brainstore-startup-gate_test.yaml b/braintrust/tests/brainstore-startup-gate_test.yaml index 758da25..8a70f0f 100644 --- a/braintrust/tests/brainstore-startup-gate_test.yaml +++ b/braintrust/tests/brainstore-startup-gate_test.yaml @@ -2,132 +2,49 @@ suite: test Brainstore API startup gate templates: - api-deployment.yaml - api-configmap.yaml - - brainstore-startup-gate-configmap.yaml - brainstore-startup-gate-rbac.yaml - api-serviceaccount.yaml - brainstore-reader-deployment.yaml - brainstore-reader-configmap.yaml tests: - - it: should gate API startup by default - template: api-deployment.yaml - asserts: - - equal: - path: spec.template.spec.initContainers[0].name - value: wait-for-brainstore - - isNotNull: - path: spec.template.metadata.annotations["checksum/brainstore-startup-gate"] - - equal: - path: spec.template.spec.automountServiceAccountToken - value: false - - it: should leave API startup unchanged when explicitly disabled template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: false + deploymentOrdering.enabled: false asserts: - isNull: path: spec.template.spec.initContainers - - isNull: - path: spec.template.metadata.annotations["checksum/brainstore-startup-gate"] - isNull: path: spec.template.spec.automountServiceAccountToken - it: should not create gate resources when disabled template: brainstore-startup-gate-rbac.yaml set: - api.brainstoreStartupGate.enabled: false - asserts: - - hasDocuments: - count: 0 - - - it: should not create the helper ConfigMap when disabled - template: brainstore-startup-gate-configmap.yaml - set: - api.brainstoreStartupGate.enabled: false + deploymentOrdering.enabled: false asserts: - hasDocuments: count: 0 - - it: should gate reused values that do not include the new setting - template: api-deployment.yaml - set: - api.brainstoreStartupGate: null - asserts: - - equal: - path: spec.template.spec.initContainers[0].name - value: wait-for-brainstore - - - it: should keep the release floor in the API Pod environment - template: api-deployment.yaml - set: - api.brainstoreStartupGate.enabled: true - api.image.tag: v2.15.0 - brainstore.image.tag: v2.15.0 - asserts: - - equal: - path: spec.template.spec.initContainers[0].name - value: wait-for-brainstore - - equal: - path: spec.progressDeadlineSeconds - value: 1500 - - equal: - path: spec.template.spec.automountServiceAccountToken - value: false - - contains: - path: spec.template.spec.initContainers[0].env - content: - name: BRAINSTORE_MINIMUM_VERSION - value: "2.15.0" - - contains: - path: spec.template.spec.initContainers[0].env - content: - name: BRAINSTORE_GATE_TIMEOUT_SECONDS - value: "1200" - - contains: - path: spec.template.spec.initContainers[0].env - content: - name: BRAINSTORE_GATE_POLL_SECONDS - value: "15" - - equal: - path: spec.template.spec.initContainers[0].env[0] - value: - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - equal: - path: spec.template.spec.initContainers[0].securityContext.runAsNonRoot - value: true - - equal: - path: spec.template.spec.initContainers[0].securityContext.readOnlyRootFilesystem - value: true - - equal: - path: spec.template.spec.volumes[1].projected.sources[0].serviceAccountToken.expirationSeconds - value: 3600 - - isNotNull: - path: spec.template.metadata.annotations["checksum/brainstore-startup-gate"] - - it: should allow an explicit minimum floor and mirrored helper image template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true - api.brainstoreStartupGate.minimumVersion: v2.15.0 - api.brainstoreStartupGate.image: registry.example/python:3.13 + deploymentOrdering.enabled: true + deploymentOrdering.minimumVersion: v2.15.0 + deploymentOrdering.image: registry.example/python:3.13 brainstore.image.tag: v2.15.1 asserts: - equal: - path: spec.template.spec.initContainers[0].image + path: spec.template.spec.initContainers[1].image value: registry.example/python:3.13 - contains: - path: spec.template.spec.initContainers[0].env + path: spec.template.spec.initContainers[1].env content: name: BRAINSTORE_MINIMUM_VERSION value: "2.15.0" - - it: should mount gate credentials only into the init container template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true api.tmpVolume.enabled: true api.extraContainers: - name: metrics-sidecar @@ -139,10 +56,28 @@ tests: - equal: path: spec.template.spec.automountServiceAccountToken value: false + - equal: + path: spec.template.spec.initContainers[0].securityContext.runAsNonRoot + value: true + - equal: + path: spec.template.spec.initContainers[0].securityContext.readOnlyRootFilesystem + value: true + - equal: + path: spec.template.spec.initContainers[1].securityContext.runAsNonRoot + value: true + - equal: + path: spec.template.spec.initContainers[1].securityContext.readOnlyRootFilesystem + value: true - contains: path: spec.template.spec.initContainers[0].volumeMounts content: - name: brainstore-gate-credentials + name: migration-gate-credentials + mountPath: /var/run/migration-gate + readOnly: true + - contains: + path: spec.template.spec.initContainers[1].volumeMounts + content: + name: migration-gate-credentials mountPath: /var/run/brainstore-gate readOnly: true - equal: @@ -159,48 +94,30 @@ tests: - it: should infer the API floor from a stable tag with a digest template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true api.image.tag: v2.15.0@sha256:abc asserts: - contains: - path: spec.template.spec.initContainers[0].env + path: spec.template.spec.initContainers[1].env content: name: BRAINSTORE_MINIMUM_VERSION value: "2.15.0" - - it: should gate each API pool even when a pool override tries to disable it - template: api-deployment.yaml - set: - api.brainstoreStartupGate.enabled: true - api.workloadIsolation.enabled: true - api.workloadIsolation.background.brainstoreStartupGate.enabled: false - documentIndex: 2 - asserts: - - equal: - path: metadata.name - value: braintrust-api-background - - equal: - path: spec.template.spec.initContainers[0].name - value: wait-for-brainstore - - equal: - path: spec.template.spec.automountServiceAccountToken - value: false - - it: should gate the ingest pool template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true api.workloadIsolation.enabled: true documentIndex: 1 asserts: - equal: - path: spec.template.spec.initContainers[0].name + path: spec.template.spec.initContainers[1].name value: wait-for-brainstore - it: should grant only named deployment reads and namespace pod lists template: brainstore-startup-gate-rbac.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true brainstore.reader.name: custom-reader documentIndex: 0 asserts: @@ -211,7 +128,7 @@ tests: value: - apiGroups: [apps] resources: [deployments] - resourceNames: [custom-reader, brainstore-writer, brainstore-fastreader] + resourceNames: [custom-reader, brainstore-writer, brainstore-fastreader, brainstore-automationwriter] verbs: [get] - apiGroups: [""] resources: [pods] @@ -223,7 +140,7 @@ tests: name: gate-test namespace: install-namespace set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true global.createNamespace: true global.namespace: workload-namespace api.workloadIsolation.enabled: true @@ -243,32 +160,24 @@ tests: name: braintrust-api namespace: workload-namespace - - it: should mount helper code without changing Brainstore startup - template: brainstore-reader-deployment.yaml - set: - api.brainstoreStartupGate.enabled: true - asserts: - - isNull: - path: spec.template.spec.initContainers - - it: should reject opaque API tags when no minimum release can be inferred template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true api.image.tag: deadbeef + deploymentOrdering.migrationImage: public.ecr.aws/braintrust/standalone-api:v2.23.0 asserts: - - failedTemplate: - errorMessage: api.brainstoreStartupGate requires a stable API release tag or an explicit stable minimumVersion + - failedTemplate: {} - it: should protect an API-only image bump while Brainstore is still old template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true api.image.tag: v2.15.0 brainstore.image.tag: v2.14.0 asserts: - contains: - path: spec.template.spec.initContainers[0].env + path: spec.template.spec.initContainers[1].env content: name: BRAINSTORE_MINIMUM_VERSION value: "2.15.0" @@ -276,89 +185,71 @@ tests: - it: should derive the release floor independently for overridden API pools template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true api.workloadIsolation.enabled: true api.image.tag: v2.14.0 api.workloadIsolation.background.image.tag: v2.15.0 documentIndex: 2 asserts: - contains: - path: spec.template.spec.initContainers[0].env + path: spec.template.spec.initContainers[1].env content: name: BRAINSTORE_MINIMUM_VERSION value: "2.15.0" - - it: should reject invalid timing - template: brainstore-startup-gate-rbac.yaml - set: - api.brainstoreStartupGate.enabled: true - api.brainstoreStartupGate.timeoutSeconds: 0 - asserts: - - failedTemplate: - errorMessage: api.brainstoreStartupGate timing values must be positive integers - - it: should pass timing overrides to the checker and extend the progress deadline template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true - api.brainstoreStartupGate.timeoutSeconds: 1800 - api.brainstoreStartupGate.pollIntervalSeconds: 30 + deploymentOrdering.enabled: true + deploymentOrdering.timeoutSeconds: 1800 + deploymentOrdering.pollIntervalSeconds: 30 asserts: - contains: - path: spec.template.spec.initContainers[0].env + path: spec.template.spec.initContainers[1].env content: name: BRAINSTORE_GATE_TIMEOUT_SECONDS value: "1800" - contains: - path: spec.template.spec.initContainers[0].env + path: spec.template.spec.initContainers[1].env content: name: BRAINSTORE_GATE_POLL_SECONDS value: "30" - equal: path: spec.progressDeadlineSeconds - value: 2100 - - - it: should reject an insufficient confirmation budget - template: brainstore-startup-gate-rbac.yaml - set: - api.brainstoreStartupGate.enabled: true - api.brainstoreStartupGate.timeoutSeconds: 15 - asserts: - - failedTemplate: - errorMessage: api.brainstoreStartupGate timeoutSeconds must exceed pollIntervalSeconds for two observations + value: 3900 - it: should reject an API rollout that removes old serving capacity template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true api.strategy.rollingUpdate.maxUnavailable: 1 asserts: - failedTemplate: - errorMessage: "api.brainstoreStartupGate requires RollingUpdate with maxUnavailable: 0 for every API pool" + errorMessage: "deploymentOrdering requires RollingUpdate with maxUnavailable: 0 for every API pool" - it: should reject unsafe strategy overrides in isolated pools template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true api.workloadIsolation.enabled: true api.workloadIsolation.background.strategy.type: Recreate asserts: - failedTemplate: - errorMessage: "api.brainstoreStartupGate requires RollingUpdate with maxUnavailable: 0 for every API pool" + errorMessage: "deploymentOrdering requires RollingUpdate with maxUnavailable: 0 for every API pool" - it: should reject a rollout with no room for gated replacement pods template: api-deployment.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true api.strategy.rollingUpdate.maxSurge: 0 asserts: - failedTemplate: - errorMessage: api.brainstoreStartupGate requires nonzero maxSurge for every API pool + errorMessage: deploymentOrdering requires nonzero maxSurge for every API pool - - it: should not leak gate labels into Brainstore or mutate global labels - template: brainstore-startup-gate-configmap.yaml + - it: should preserve service account labels after gate rendering + template: api-serviceaccount.yaml set: - api.brainstoreStartupGate.enabled: true + deploymentOrdering.enabled: true global.labels.shared: global api.labels.api-only: api brainstore.reader.labels.reader-only: reader @@ -366,22 +257,57 @@ tests: - equal: path: metadata.labels.shared value: global - - equal: - path: metadata.labels["api-only"] - value: api - isNull: path: metadata.labels["reader-only"] - - it: should preserve service account labels after gate rendering - template: api-serviceaccount.yaml + - it: binds optional application identities for direct rollout reads + template: brainstore-startup-gate-rbac.yaml + documentIndex: 1 + values: + - __fixtures__/loop-runtime-values.yaml + release: + name: test + namespace: workload set: - api.brainstoreStartupGate.enabled: true - global.labels.shared: global - api.labels.api-only: api - brainstore.reader.labels.reader-only: reader + aiGateway.enabled: true + aiGateway.serviceAccount.name: gateway-account + asserts: + - contains: + path: subjects + content: {kind: ServiceAccount, name: gateway-account, namespace: workload} + - contains: + path: subjects + content: {kind: ServiceAccount, name: braintrust-loop-runtime, namespace: workload} + + - it: requires an explicit compatibility assertion for opaque Brainstore images + template: brainstore-reader-deployment.yaml + set: + brainstore.image.tag: deadbeef + deploymentOrdering.migrationImage: public.ecr.aws/braintrust/standalone-api:v2.23.0 + asserts: + - failedTemplate: {} + + - it: reserves the actual stable Brainstore version against annotation overrides + template: brainstore-reader-deployment.yaml + set: + brainstore.image.tag: v2.22.0@sha256:abc + brainstore.reader.annotations.pod: + braintrust.dev/brainstore-version: "9.0.0" + deploymentOrdering.minimumVersion: "2.21.0" asserts: - equal: - path: metadata.labels.shared - value: global - - isNull: - path: metadata.labels["reader-only"] + path: spec.template.metadata.annotations["braintrust.dev/brainstore-version"] + value: "2.22.0" + + - it: reserves the explicit compatibility assertion for opaque Brainstore builds + template: brainstore-reader-deployment.yaml + set: + brainstore.image.tag: deadbeef + deploymentOrdering.minimumVersion: "2.22.0" + deploymentOrdering.migrationImage: public.ecr.aws/braintrust/standalone-api:release-bundle + brainstore.reader.annotations.pod: + braintrust.dev/brainstore-version: "9.0.0" + asserts: + - equal: + path: spec.template.metadata.annotations["braintrust.dev/brainstore-version"] + value: "2.22.0" diff --git a/braintrust/tests/brainstore-writer_test.yaml b/braintrust/tests/brainstore-writer_test.yaml index d2af988..8a8c82a 100644 --- a/braintrust/tests/brainstore-writer_test.yaml +++ b/braintrust/tests/brainstore-writer_test.yaml @@ -2,6 +2,9 @@ suite: test Brainstore Writer deployment template templates: - brainstore-writer-deployment.yaml - brainstore-writer-configmap.yaml +# Test workload details independently of release ordering. +set: + deploymentOrdering.enabled: false tests: - it: should render Brainstore Writer deployment with default values template: brainstore-writer-deployment.yaml diff --git a/braintrust/tests/labels-merge-serviceaccounts_test.yaml b/braintrust/tests/labels-merge-serviceaccounts_test.yaml index 8e661f9..435a65f 100644 --- a/braintrust/tests/labels-merge-serviceaccounts_test.yaml +++ b/braintrust/tests/labels-merge-serviceaccounts_test.yaml @@ -3,6 +3,7 @@ templates: - api-serviceaccount.yaml - loop-runtime-serviceaccount.yaml - brainstore-serviceaccount.yaml + - ai-gateway-serviceaccount.yaml tests: # These tests verify the deepCopy fix for service accounts. # Without deepCopy, labels would bleed between components due to merge mutation. @@ -65,3 +66,41 @@ tests: value: shared: global-sa loop-specific: loop-sa + + - it: keeps Gateway account labels and annotations scoped to that account + template: ai-gateway-serviceaccount.yaml + set: &gatewayIdentityLabels + aiGateway.enabled: true + global.labels.shared: global-sa + global.labels.tier: global-tier + aiGateway.labels.tier: gateway-tier + aiGateway.labels.gateway-only: gateway-sa + api.labels.api-only: api-sa + aiGateway.annotations.serviceaccount: + example.com/identity: gateway + aiGateway.annotations.pod: + example.com/pod-only: gateway-pod + asserts: + - equal: + path: metadata.labels + value: + shared: global-sa + tier: gateway-tier + gateway-only: gateway-sa + - equal: + path: metadata.annotations + value: + example.com/identity: gateway + + - it: does not leak Gateway identity metadata into the API account + template: api-serviceaccount.yaml + set: *gatewayIdentityLabels + asserts: + - equal: + path: metadata.labels + value: + shared: global-sa + tier: global-tier + api-only: api-sa + - isNull: + path: metadata.annotations["example.com/identity"] diff --git a/braintrust/tests/legacy-startup-gate_test.yaml b/braintrust/tests/legacy-startup-gate_test.yaml new file mode 100644 index 0000000..a90c4b7 --- /dev/null +++ b/braintrust/tests/legacy-startup-gate_test.yaml @@ -0,0 +1,172 @@ +suite: retain existing legacy Brainstore startup gate ConfigMaps +templates: + - brainstore-startup-gate-configmap.yaml + - api-configmap.yaml +release: + name: legacy + namespace: release-ns +kubernetesProvider: + scheme: + v1/ConfigMap: + gvr: + version: v1 + resource: configmaps + namespaced: true + objects: + # Keep the fake provider active even when the requested legacy gate is absent. + - apiVersion: v1 + kind: ConfigMap + metadata: + name: unrelated-configmap + namespace: release-ns +tests: + - it: preserves every original text and binary entry instead of substituting the new gate + template: brainstore-startup-gate-configmap.yaml + kubernetesProvider: &existingGate + objects: + - apiVersion: v1 + kind: ConfigMap + metadata: + name: legacy-brainstore-startup + namespace: release-ns + data: &legacyData + gate.py: | + # Original legacy script; it has no rollout-revision arguments. + print("keep original script: {{ do not evaluate }}") + other-key: "literal: value\nwith a trailing newline\n" + empty-key: "" + binaryData: &legacyBinaryData + original.bin: AAEC/w== + empty.bin: "" + set: + global.labels.shared: global + global.labels.global-only: retained + api.labels.shared: api + api.annotations.configmap: + operator-note: retained + helm.sh/resource-policy: delete + asserts: + - hasDocuments: + count: 1 + - isKind: + of: ConfigMap + - equal: + path: metadata.name + value: legacy-brainstore-startup + - equal: + path: metadata.namespace + value: release-ns + - equal: + path: metadata.labels + value: + shared: api + global-only: retained + - equal: + path: metadata.annotations + value: + operator-note: retained + helm.sh/resource-policy: keep + - equal: + path: data + value: *legacyData + - equal: + path: binaryData + value: *legacyBinaryData + + - it: retains the original gate even when deployment ordering is disabled + template: brainstore-startup-gate-configmap.yaml + kubernetesProvider: *existingGate + set: + deploymentOrdering.enabled: false + asserts: + - hasDocuments: + count: 1 + - equal: + path: data + value: *legacyData + - equal: + path: binaryData + value: *legacyBinaryData + - equal: + path: metadata.annotations["helm.sh/resource-policy"] + value: keep + + - it: looks up the actual workload namespace rather than the release namespace + template: brainstore-startup-gate-configmap.yaml + set: + global.createNamespace: true + global.namespace: workload-ns + kubernetesProvider: + objects: + - apiVersion: v1 + kind: ConfigMap + metadata: + name: legacy-brainstore-startup + namespace: workload-ns + data: + gate.py: original workload namespace script + - apiVersion: v1 + kind: ConfigMap + metadata: + name: legacy-brainstore-startup + namespace: release-ns + data: + gate.py: wrong release namespace script + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.namespace + value: workload-ns + - equal: + path: data + value: + gate.py: original workload namespace script + + - it: does not create a gate on fresh installs without an existing object + template: brainstore-startup-gate-configmap.yaml + asserts: + - hasDocuments: + count: 0 + + - it: does not retain a same-named object outside the workload namespace + template: brainstore-startup-gate-configmap.yaml + kubernetesProvider: *existingGate + set: + global.createNamespace: true + global.namespace: workload-ns + asserts: + - hasDocuments: + count: 0 + + - it: preserves binary-only objects without inventing text data + template: brainstore-startup-gate-configmap.yaml + kubernetesProvider: + objects: + - apiVersion: v1 + kind: ConfigMap + metadata: + name: legacy-brainstore-startup + namespace: release-ns + binaryData: *legacyBinaryData + asserts: + - hasDocuments: + count: 1 + - isNull: + path: data + - equal: + path: binaryData + value: *legacyBinaryData + + - it: does not leak retention annotations into application ConfigMaps + template: api-configmap.yaml + kubernetesProvider: *existingGate + set: + api.annotations.configmap: + operator-note: preserved + asserts: + - equal: + path: metadata.annotations.operator-note + value: preserved + - isNull: + path: metadata.annotations["helm.sh/resource-policy"] diff --git a/braintrust/tests/migration-image_test.yaml b/braintrust/tests/migration-image_test.yaml new file mode 100644 index 0000000..ffc10ac --- /dev/null +++ b/braintrust/tests/migration-image_test.yaml @@ -0,0 +1,413 @@ +suite: migration bundle selection across serving images +templates: + - migration-job.yaml + - api-configmap.yaml +release: + name: test + namespace: workload + revision: 7 +set: + api.image.repository: registry.example/standalone-api + api.image.tag: v2.21.0 + brainstore.image.repository: registry.example/brainstore + brainstore.image.tag: v2.22.0 +tests: + - it: uses standalone API scripts at the newer Brainstore release + template: migration-job.yaml + documentIndex: 0 + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.22.0 + - matchRegex: + path: metadata.name + pattern: '^test-migrate-[a-f0-9]{16}$' + - matchRegex: + path: spec.template.spec.containers[0].envFrom[0].configMapRef.name + pattern: '^test-migrate-[a-f0-9]{16}-config$' + + - it: preserves the repository and full pin of a newer API image + template: migration-job.yaml + documentIndex: 0 + set: + api.image.repository: mirror.example:5000/custom/api + api.image.tag: v2.24.0@sha256:api + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: mirror.example:5000/custom/api:v2.24.0@sha256:api + + - it: accepts stable API tags without a v prefix + template: migration-job.yaml + documentIndex: 0 + set: + api.image.tag: 2.24.0 + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:2.24.0 + + - it: drops the Brainstore digest when selecting its release in the API repository + template: migration-job.yaml + documentIndex: 0 + set: + brainstore.image.tag: v2.24.0@sha256:brainstore + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.24.0 + + - it: preserves a Brainstore stable tag without its v prefix or digest + template: migration-job.yaml + documentIndex: 0 + set: + brainstore.image.tag: 2.24.0@sha256:brainstore + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:2.24.0 + + - it: compares release components numerically rather than lexicographically + template: migration-job.yaml + documentIndex: 0 + set: + api.image.tag: v2.9.0 + brainstore.image.tag: v2.22.0 + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.22.0 + + - it: keeps the API digest when API and Brainstore releases are equal + template: migration-job.yaml + documentIndex: 0 + set: + api.image.tag: v2.22.0@sha256:api + brainstore.image.tag: 2.22.0@sha256:brainstore + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.22.0@sha256:api + + - it: selects a newer isolated ingest pool including its custom repository and digest + template: migration-job.yaml + documentIndex: 0 + set: + api.workloadIsolation.enabled: true + api.workloadIsolation.ingest.image.repository: mirror.example:5000/ingest-api + api.workloadIsolation.ingest.image.tag: v2.24.0@sha256:ingest + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: mirror.example:5000/ingest-api:v2.24.0@sha256:ingest + + - it: considers the background pool even when ingest is also newer than the shared API + template: migration-job.yaml + documentIndex: 0 + set: + api.workloadIsolation.enabled: true + api.workloadIsolation.ingest.image.tag: v2.24.0 + api.workloadIsolation.background.image.repository: registry.example/background-api + api.workloadIsolation.background.image.tag: 2.25.0@sha256:background + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/background-api:2.25.0@sha256:background + + - it: uses the shared API repository when Brainstore is newer than every isolated pool + template: migration-job.yaml + documentIndex: 0 + set: + api.workloadIsolation.enabled: true + api.workloadIsolation.background.image.repository: registry.example/background-api + api.workloadIsolation.background.image.tag: v2.25.0 + brainstore.image.tag: v2.26.0@sha256:brainstore + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.26.0 + + - it: ignores image overrides for API pools that are not rendered + template: migration-job.yaml + documentIndex: 0 + set: + api.workloadIsolation.enabled: false + api.workloadIsolation.ingest.image.tag: v3.0.0 + api.workloadIsolation.background.image.tag: deadbeef + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.22.0 + + - it: blocks a newer API major before executing migrations + template: migration-job.yaml + set: + api.image.tag: v3.0.0 + asserts: + - failedTemplate: &mixed-major + errorMessage: deploymentOrdering cannot migrate API and Brainstore service images from different major releases + + - it: blocks a newer Brainstore major before executing migrations + template: migration-job.yaml + set: + brainstore.image.tag: v3.0.0@sha256:brainstore + asserts: + - failedTemplate: *mixed-major + + - it: does not let an explicit opaque bundle bypass incompatible API pool majors + template: migration-job.yaml + set: + api.workloadIsolation.enabled: true + api.workloadIsolation.background.image.tag: v3.0.0 + deploymentOrdering.migrationImage: registry.example/migration-api:custom + asserts: + - failedTemplate: *mixed-major + + - it: requires a bundle for a SHA API even with an explicit release floor + template: migration-job.yaml + set: + api.image.tag: deadbeef + deploymentOrdering.minimumVersion: v2.22.0 + asserts: + - failedTemplate: &opaque-bundle + errorMessage: deploymentOrdering.migrationImage is required for opaque API or Brainstore service image tags; minimumVersion cannot select a migration bundle + + - it: requires a bundle for a SHA Brainstore instead of borrowing the floor's release + template: migration-job.yaml + set: + brainstore.image.tag: 0123456789abcdef0123456789abcdef0123456789 + deploymentOrdering.minimumVersion: v2.25.0 + asserts: + - failedTemplate: *opaque-bundle + + - it: handles an all-digit numeric API SHA without a semver parsing error + template: migration-job.yaml + set: + api.image.tag: 1234567 + deploymentOrdering.minimumVersion: v2.22.0 + asserts: + - failedTemplate: *opaque-bundle + + - it: requires a bundle for prerelease serving tags + template: migration-job.yaml + set: + api.image.tag: v2.24.0-rc.1 + deploymentOrdering.minimumVersion: v2.22.0 + asserts: + - failedTemplate: *opaque-bundle + + - it: requires a bundle for custom serving tags with a digest + template: migration-job.yaml + set: + brainstore.image.tag: customer-build@sha256:brainstore + deploymentOrdering.minimumVersion: v2.22.0 + asserts: + - failedTemplate: *opaque-bundle + + - it: requires a bundle if any isolated API pool is opaque + template: migration-job.yaml + set: + api.workloadIsolation.enabled: true + api.workloadIsolation.ingest.image.tag: deadbeef + api.workloadIsolation.background.image.tag: v2.25.0 + deploymentOrdering.minimumVersion: v2.22.0 + asserts: + - failedTemplate: *opaque-bundle + + - it: accepts an explicit digest-only standalone API bundle for opaque services + template: migration-job.yaml + documentIndex: 0 + set: + api.image.tag: 0123456789abcdef0123456789abcdef0123456789 + brainstore.image.tag: deadbeef + deploymentOrdering.minimumVersion: v2.22.0 + deploymentOrdering.migrationImage: mirror.example:5000/standalone-api@sha256:bundle + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: mirror.example:5000/standalone-api@sha256:bundle + - matchRegex: + path: metadata.name + pattern: '^test-migrate-[a-f0-9]{16}$' + + - it: accepts an explicit prerelease bundle as the operator's compatibility assertion + template: migration-job.yaml + documentIndex: 0 + set: + api.image.tag: 1234567 + brainstore.image.tag: customer-build + deploymentOrdering.minimumVersion: v2.22.0 + deploymentOrdering.migrationImage: registry.example/standalone-api:v2.24.0-rc.1@sha256:bundle + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.24.0-rc.1@sha256:bundle + + - it: accepts an opaque bundle for stable services without inventing a floor requirement + template: migration-job.yaml + documentIndex: 0 + set: + deploymentOrdering.migrationImage: registry.example/standalone-api:customer-migrations + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:customer-migrations + + - it: preserves a known explicit bundle tag and digest at the required release + template: migration-job.yaml + documentIndex: 0 + set: + deploymentOrdering.migrationImage: mirror.example:5000/standalone-api:2.22.0@sha256:bundle + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: mirror.example:5000/standalone-api:2.22.0@sha256:bundle + + - it: rejects a known explicit bundle older than Brainstore + template: migration-job.yaml + set: + deploymentOrdering.migrationImage: registry.example/standalone-api:v2.21.0@sha256:bundle + asserts: + - failedTemplate: + errorMessage: deploymentOrdering.migrationImage release v2.21.0 is older than required release 2.22.0 + + - it: rejects a known explicit bundle older than the newest isolated API pool + template: migration-job.yaml + set: + api.workloadIsolation.enabled: true + api.workloadIsolation.background.image.tag: v2.25.0 + deploymentOrdering.migrationImage: registry.example/standalone-api:v2.24.0 + asserts: + - failedTemplate: + errorMessage: deploymentOrdering.migrationImage release v2.24.0 is older than required release 2.25.0 + + - it: rejects a known explicit bundle older than the explicit floor for opaque services + template: migration-job.yaml + set: + api.image.tag: deadbeef + deploymentOrdering.minimumVersion: v2.25.0 + deploymentOrdering.migrationImage: registry.example/standalone-api:v2.24.0 + asserts: + - failedTemplate: + errorMessage: deploymentOrdering.migrationImage release v2.24.0 is older than required release 2.25.0 + + - it: accepts a known explicit bundle at the floor for opaque services + template: migration-job.yaml + documentIndex: 0 + set: + api.image.tag: deadbeef + brainstore.image.tag: customer-build + deploymentOrdering.minimumVersion: v2.25.0 + deploymentOrdering.migrationImage: registry.example/standalone-api:v2.25.0 + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.25.0 + + - it: still requires a release floor for opaque services when a bundle is explicit + template: migration-job.yaml + set: + brainstore.image.tag: deadbeef + deploymentOrdering.migrationImage: registry.example/standalone-api:customer-migrations + asserts: + - failedTemplate: + errorMessage: deploymentOrdering.minimumVersion is required for opaque API or Brainstore service image tags + + - it: validates an explicit floor before comparing a stable bundle against it + template: migration-job.yaml + set: + deploymentOrdering.minimumVersion: not-a-version + deploymentOrdering.migrationImage: registry.example/standalone-api:v2.24.0 + asserts: + - failedTemplate: + errorMessage: deploymentOrdering.minimumVersion must be a stable release version + + - it: leaves an explicitly untagged standalone API reference to the operator's assertion + template: migration-job.yaml + documentIndex: 0 + set: + deploymentOrdering.migrationImage: registry.example/standalone-api + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api + + - it: rejects a non-string migration image instead of silently ignoring zero + template: migration-job.yaml + set: + deploymentOrdering.migrationImage: 0 + asserts: + - failedTemplate: + errorMessage: deploymentOrdering.migrationImage must be a full standalone-API image reference + + - it: treats a null migration image as automatic selection + template: migration-job.yaml + documentIndex: 0 + set: + deploymentOrdering.migrationImage: null + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.22.0 + + - it: selects a bundle when reused values omit the ordering map + template: migration-job.yaml + documentIndex: 0 + set: + deploymentOrdering: null + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.22.0 + + - it: ignores unused opaque images invalid overrides and floors when ordering is disabled + template: migration-job.yaml + set: + deploymentOrdering.enabled: false + deploymentOrdering.minimumVersion: not-a-version + deploymentOrdering.migrationImage: 0 + api.image.tag: deadbeef + brainstore.image.tag: customer-build + asserts: + - hasDocuments: + count: 0 + + - it: does not reject unused cross-major combinations when ordering is disabled + template: migration-job.yaml + set: + deploymentOrdering.enabled: false + api.image.tag: v3.0.0 + asserts: + - hasDocuments: + count: 0 + + - it: retains an immutable configuration snapshot for an opaque pinned migration bundle + template: migration-job.yaml + documentIndex: 1 + set: + api.image.tag: customer-api + brainstore.image.tag: customer-brainstore + deploymentOrdering.minimumVersion: v2.22.0 + deploymentOrdering.migrationImage: mirror.example:5000/standalone-api@sha256:bundle + asserts: + - equal: + path: immutable + value: true + - equal: + path: metadata.annotations["helm.sh/resource-policy"] + value: keep + - matchRegex: + path: metadata.name + pattern: '^test-migrate-[a-f0-9]{16}-config$' + + - it: keeps migration execution independent of the waiter helper image + template: migration-job.yaml + documentIndex: 0 + set: + deploymentOrdering.image: mirror.example/python:3.13 + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/standalone-api:v2.22.0 + - isNull: + path: spec.template.spec.initContainers diff --git a/braintrust/tests/migrations_test.yaml b/braintrust/tests/migrations_test.yaml new file mode 100644 index 0000000..6d50b6d --- /dev/null +++ b/braintrust/tests/migrations_test.yaml @@ -0,0 +1,639 @@ +suite: content-addressed migration Job and stable Deployment dependencies +templates: + - migration-job.yaml + - migration-gate.yaml + - api-deployment.yaml + - api-configmap.yaml + - brainstore-reader-deployment.yaml + - brainstore-reader-configmap.yaml + - brainstore-writer-deployment.yaml + - brainstore-writer-configmap.yaml + - brainstore-fastreader-deployment.yaml + - brainstore-fastreader-configmap.yaml + - brainstore-automationwriter-deployment.yaml + - brainstore-automationwriter-configmap.yaml + - ai-gateway-deployment.yaml + - ai-gateway-configmap.yaml + - loop-runtime-deployment.yaml + - loop-runtime-configmap.yaml + - api-serviceaccount.yaml +release: + name: test + namespace: workload + revision: 7 +tests: + - it: runs a regular bounded migration Job using the newest standalone API bundle + template: migration-job.yaml + documentIndex: 0 + set: + api.image.repository: registry.example/custom-api + api.image.tag: v2.15.0 + asserts: + - isKind: + of: Job + - matchRegex: + path: metadata.name + pattern: '^test-migrate-[a-f0-9]{16}$' + - equal: + path: metadata.annotations["helm.sh/resource-policy"] + value: keep + - isNull: + path: metadata.annotations["helm.sh/hook"] + - isNull: + path: spec.ttlSecondsAfterFinished + - equal: + path: spec.activeDeadlineSeconds + value: 1200 + - equal: + path: spec.backoffLimit + value: 2 + - equal: + path: spec.template.spec.restartPolicy + value: Never + - equal: + path: spec.template.spec.containers[0].command + value: [/braintrust/venv/bin/python, /braintrust/api-schema/lambda_function.py, --execute] + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/custom-api:v2.23.0 + - isNull: + path: spec.template.spec.initContainers + - isNull: + path: spec.template.spec.containers[0].readinessProbe + - isNull: + path: spec.template.spec.containers[0].livenessProbe + - isNull: + path: spec.template.spec.containers[0].ports + - notEqual: + path: spec.template.metadata.labels.app + value: braintrust-api + - equal: + path: spec.template.spec.automountServiceAccountToken + value: false + - matchRegex: + path: spec.template.spec.containers[0].envFrom[0].configMapRef.name + pattern: '^test-migrate-[a-f0-9]{16}-config$' + - matchRegex: + path: spec.template.metadata.annotations["checksum/config"] + pattern: '^[a-f0-9]{64}$' + + - it: retains an immutable configuration snapshot without copying Secrets + template: migration-job.yaml + documentIndex: 1 + set: + global.orgName: migration-org + asserts: + - isKind: + of: ConfigMap + - matchRegex: + path: metadata.name + pattern: '-migrate-[a-f0-9]{16}-config$' + - equal: + path: data.ORG_NAME + value: migration-org + - equal: + path: metadata.annotations["helm.sh/resource-policy"] + value: keep + - equal: + path: immutable + value: true + - isNull: + path: data.PG_URL + + - it: waits for migrations before directly checking the Brainstore rollout + template: api-deployment.yaml + asserts: + - equal: + path: spec.template.spec.initContainers[0].name + value: wait-for-migrations + - equal: + path: spec.template.spec.initContainers[1].name + value: wait-for-brainstore + - equal: + path: spec.template.spec.initContainers[0].env[1] + value: {name: ORDERING_CONFIGMAP, value: test-9f86d081-ordering} + - contains: + path: spec.template.spec.volumes + content: {name: migration-gate-code, configMap: {name: test-9f86d081-ordering}} + - equal: + path: spec.template.spec.automountServiceAccountToken + value: false + + - it: restricts workload gates to reading the desired-state ConfigMap and exact Job + template: migration-gate.yaml + documentIndex: 1 + asserts: + - isKind: + of: Role + - equal: + path: rules[0].apiGroups + value: [batch] + - equal: + path: rules[0].resources + value: [jobs] + - equal: + path: rules[0].verbs + value: [get] + - lengthEqual: + path: rules + count: 2 + - matchRegex: + path: rules[0].resourceNames[0] + pattern: '-migrate-[a-f0-9]{16}$' + - lengthEqual: + path: rules[0].resourceNames + count: 1 + - equal: + path: rules[1].apiGroups + value: [""] + - equal: + path: rules[1].resources + value: [configmaps] + - equal: + path: rules[1].verbs + value: [get] + - equal: + path: rules[1].resourceNames + value: [test-9f86d081-ordering] + + - it: binds all pool identities and the Brainstore account in the actual namespace + template: migration-gate.yaml + documentIndex: 2 + set: + global.createNamespace: true + global.namespace: actual-namespace + api.workloadIsolation.enabled: true + api.workloadIsolation.ingest.serviceAccount.name: ingest + api.workloadIsolation.background.serviceAccount.name: background + asserts: + - equal: + path: metadata.namespace + value: actual-namespace + - contains: + path: subjects + content: {kind: ServiceAccount, name: ingest, namespace: actual-namespace} + - contains: + path: subjects + content: {kind: ServiceAccount, name: background, namespace: actual-namespace} + - contains: + path: subjects + content: {kind: ServiceAccount, name: brainstore, namespace: actual-namespace} + + - it: inherits API environment CA security scheduling and volumes without sidecars + template: migration-job.yaml + documentIndex: 0 + set: + api.customCA.enabled: true + api.tmpVolume.enabled: true + api.nodeSelector.migration-node: "yes" + api.securityContext.readOnlyRootFilesystem: true + api.extraEnvVars: + - {name: MIGRATION_VERSION_TABLE, value: custom_version} + api.extraContainers: + - {name: forever-sidecar, image: busybox} + asserts: + - lengthEqual: + path: spec.template.spec.containers + count: 1 + - equal: + path: spec.template.spec.nodeSelector.migration-node + value: "yes" + - equal: + path: spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem + value: true + - contains: + path: spec.template.spec.containers[0].env + content: {name: MIGRATION_VERSION_TABLE, value: custom_version} + - contains: + path: spec.template.spec.containers[0].env + content: {name: SSL_CERT_FILE, value: /etc/braintrust/runtime-ca/ca-bundle.pem} + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: {name: custom-ca-bundle, mountPath: /etc/braintrust/runtime-ca, readOnly: true} + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: {name: tmp-volume, mountPath: /tmp} + - contains: + path: spec.template.spec.containers[0].env + content: + name: PG_URL + valueFrom: + secretKeyRef: {name: braintrust-secrets, key: PG_URL} + + - it: mounts Azure CSI secrets independently of application startup + template: migration-job.yaml + documentIndex: 0 + set: + cloud: azure + azure.enableAzureKeyVaultDriver: true + azure.enableAzureContainerStorageDriver: false + azure.keyVaultName: test-vault + asserts: + - equal: + path: spec.template.metadata.labels["azure.workload.identity/use"] + value: "true" + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: {name: secrets-store-inline, mountPath: /mnt/secrets-store, readOnly: true} + - contains: + path: spec.template.spec.volumes + content: + name: secrets-store-inline + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: {secretProviderClass: test-vault} + + - it: preserves Google native workload identity + template: migration-job.yaml + documentIndex: 0 + set: + cloud: google + api.enableGcsAuth: true + api.serviceAccount.googleServiceAccount: api@example.iam.gserviceaccount.com + brainstore.serviceAccount.googleServiceAccount: brainstore@example.iam.gserviceaccount.com + asserts: + - equal: + path: spec.template.metadata.labels["gke-workload-identity/use"] + value: "true" + - equal: + path: spec.template.metadata.annotations["iam.gke.io/gcp-service-account"] + value: api@example.iam.gserviceaccount.com + + - it: does not mutate API service labels while rendering the Job + template: api-serviceaccount.yaml + set: + global.labels.shared: global + api.labels.api-only: api + brainstore.reader.labels.reader-only: reader + asserts: + - equal: + path: metadata.labels.shared + value: global + - equal: + path: metadata.labels["api-only"] + value: api + - isNull: + path: metadata.labels["braintrust.dev/migration-release"] + - isNull: + path: metadata.labels["reader-only"] + + - it: does not leak Brainstore labels into the Job + template: migration-job.yaml + documentIndex: 0 + set: + global.labels.shared: global + api.labels.api-only: api + brainstore.reader.labels.reader-only: reader + asserts: + - equal: + path: metadata.labels.shared + value: global + - equal: + path: metadata.labels["api-only"] + value: api + - isNull: + path: metadata.labels["reader-only"] + + - it: keeps the serving API image unchanged when Brainstore selects a newer bundle + template: api-deployment.yaml + set: + api.image.repository: registry.example/custom-api + api.image.tag: v2.21.0 + brainstore.image.tag: v2.22.0 + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/custom-api:v2.21.0 + - equal: + path: spec.template.spec.initContainers[0].env[1] + value: {name: ORDERING_CONFIGMAP, value: test-9f86d081-ordering} + + - it: keeps the Brainstore image and digest unchanged when selecting an API bundle + template: brainstore-reader-deployment.yaml + set: + api.image.repository: registry.example/custom-api + api.image.tag: v2.21.0 + brainstore.image.repository: registry.example/custom-brainstore + brainstore.image.tag: v2.22.0@sha256:brainstore + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/custom-brainstore:v2.22.0@sha256:brainstore + + - it: gates the reader pool + template: brainstore-reader-deployment.yaml + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.template.spec.initContainers[0].env[1]", "value": {"name": "ORDERING_CONFIGMAP", "value": "test-9f86d081-ordering"}}} + - {"equal": {"path": "spec.progressDeadlineSeconds", "value": 1800}} + + - it: gates the writer pool + template: brainstore-writer-deployment.yaml + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.template.spec.initContainers[0].env[1]", "value": {"name": "ORDERING_CONFIGMAP", "value": "test-9f86d081-ordering"}}} + - {"equal": {"path": "spec.progressDeadlineSeconds", "value": 1800}} + + - it: gates the fastreader pool + template: brainstore-fastreader-deployment.yaml + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.template.spec.initContainers[0].env[1]", "value": {"name": "ORDERING_CONFIGMAP", "value": "test-9f86d081-ordering"}}} + - {"equal": {"path": "spec.progressDeadlineSeconds", "value": 1800}} + + - it: gates the automationwriter pool + template: brainstore-automationwriter-deployment.yaml + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.template.spec.initContainers[0].env[1]", "value": {"name": "ORDERING_CONFIGMAP", "value": "test-9f86d081-ordering"}}} + - {"equal": {"path": "spec.progressDeadlineSeconds", "value": 1800}} + + - it: gates isolated API pool 0 + template: api-deployment.yaml + documentIndex: 0 + set: {"api.workloadIsolation.enabled": true} + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.template.spec.initContainers[1].name", "value": "wait-for-brainstore"}} + + - it: gates isolated API pool 1 + template: api-deployment.yaml + documentIndex: 1 + set: {"api.workloadIsolation.enabled": true} + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.template.spec.initContainers[1].name", "value": "wait-for-brainstore"}} + + - it: gates isolated API pool 2 + template: api-deployment.yaml + documentIndex: 2 + set: {"api.workloadIsolation.enabled": true} + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.template.spec.initContainers[1].name", "value": "wait-for-brainstore"}} + + - it: gates the optional AI Gateway + template: ai-gateway-deployment.yaml + set: {"aiGateway.enabled": true} + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.template.spec.automountServiceAccountToken", "value": false}} + - {"equal": {"path": "spec.template.spec.initContainers[1].name", "value": "wait-for-brainstore"}} + - {"equal": {"path": "spec.progressDeadlineSeconds", "value": 3000}} + + - it: gates the optional Loop runtime + values: + - __fixtures__/loop-runtime-values.yaml + template: loop-runtime-deployment.yaml + set: {"cloud": "google", "loopRuntime.enabled": true, "loopRuntime.serviceAccount.googleServiceAccount": "loop@example.iam.gserviceaccount.com", "brainstore.serviceAccount.googleServiceAccount": "bs@example.iam.gserviceaccount.com"} + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.template.spec.initContainers[1].name", "value": "wait-for-brainstore"}} + - {"equal": {"path": "spec.progressDeadlineSeconds", "value": 3000}} + + - it: binds the Loop runtime identity + values: + - __fixtures__/loop-runtime-values.yaml + template: migration-gate.yaml + documentIndex: 2 + set: {"cloud": "google", "loopRuntime.enabled": true, "loopRuntime.serviceAccount.googleServiceAccount": "loop@example.iam.gserviceaccount.com", "brainstore.serviceAccount.googleServiceAccount": "bs@example.iam.gserviceaccount.com"} + asserts: + - {"contains": {"path": "subjects", "content": {"kind": "ServiceAccount", "name": "braintrust-loop-runtime", "namespace": "workload"}}} + + - it: omits migration-job.yaml when disabled + template: migration-job.yaml + set: {"deploymentOrdering.enabled": false} + asserts: + - {"hasDocuments": {"count": 0}} + + - it: omits migration-gate.yaml when disabled + template: migration-gate.yaml + set: {"deploymentOrdering.enabled": false} + asserts: + - {"hasDocuments": {"count": 0}} + + - it: removes migration wait when disabled in brainstore-reader-deployment.yaml + template: brainstore-reader-deployment.yaml + set: {"deploymentOrdering.enabled": false, "aiGateway.enabled": true} + asserts: + - {"isNull": {"path": "spec.template.spec.initContainers"}} + + - it: removes migration wait when disabled in brainstore-writer-deployment.yaml + template: brainstore-writer-deployment.yaml + set: {"deploymentOrdering.enabled": false, "aiGateway.enabled": true} + asserts: + - {"isNull": {"path": "spec.template.spec.initContainers"}} + + - it: removes migration wait when disabled in brainstore-fastreader-deployment.yaml + template: brainstore-fastreader-deployment.yaml + set: {"deploymentOrdering.enabled": false, "aiGateway.enabled": true} + asserts: + - {"isNull": {"path": "spec.template.spec.initContainers"}} + + - it: removes migration wait when disabled in brainstore-automationwriter-deployment.yaml + template: brainstore-automationwriter-deployment.yaml + set: {"deploymentOrdering.enabled": false, "aiGateway.enabled": true} + asserts: + - {"isNull": {"path": "spec.template.spec.initContainers"}} + + - it: removes migration wait when disabled in ai-gateway-deployment.yaml + template: ai-gateway-deployment.yaml + set: {"deploymentOrdering.enabled": false, "aiGateway.enabled": true} + asserts: + - {"isNull": {"path": "spec.template.spec.initContainers"}} + + - it: defaults a missing or null feature map on reused values + template: api-deployment.yaml + set: {"deploymentOrdering": null} + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + + - it: merges partial reused values with defaults + template: migration-job.yaml + documentIndex: 0 + set: + deploymentOrdering: + backoffLimit: 0 + asserts: + - {"equal": {"path": "spec.backoffLimit", "value": 0}} + - {"equal": {"path": "spec.activeDeadlineSeconds", "value": 1200}} + + - it: uses custom timing and a mirrored helper image + template: brainstore-reader-deployment.yaml + set: {"deploymentOrdering.timeoutSeconds": 1800, "deploymentOrdering.pollIntervalSeconds": 10, "deploymentOrdering.image": "mirror/python:3.13"} + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].image", "value": "mirror/python:3.13"}} + - {"equal": {"path": "spec.template.spec.initContainers[0].env[2].value", "value": "1800"}} + - {"equal": {"path": "spec.template.spec.initContainers[0].env[3].value", "value": "10"}} + + + - it: rejects invalid timeoutSeconds 0 + template: migration-job.yaml + set: {"deploymentOrdering.timeoutSeconds": 0} + asserts: + - failedTemplate: + errorMessage: deploymentOrdering timing values must be positive integers + + - it: rejects invalid timeoutSeconds + template: migration-job.yaml + set: {"deploymentOrdering.timeoutSeconds": ""} + asserts: + - failedTemplate: {} + + - it: defaults a null timeoutSeconds after Helm coalesces values + template: migration-job.yaml + documentIndex: 0 + set: {deploymentOrdering.timeoutSeconds: null} + asserts: + - equal: + path: spec.activeDeadlineSeconds + value: 1200 + + - it: rejects invalid timeoutSeconds -1 + template: migration-job.yaml + set: {"deploymentOrdering.timeoutSeconds": -1} + asserts: + - failedTemplate: {} + + - it: rejects invalid pollIntervalSeconds 0 + template: migration-job.yaml + set: {"deploymentOrdering.pollIntervalSeconds": 0} + asserts: + - failedTemplate: {} + + - it: rejects invalid pollIntervalSeconds + template: migration-job.yaml + set: {"deploymentOrdering.pollIntervalSeconds": ""} + asserts: + - failedTemplate: {} + + - it: rejects invalid pollIntervalSeconds -1 + template: migration-job.yaml + set: {"deploymentOrdering.pollIntervalSeconds": -1} + asserts: + - failedTemplate: {} + + - it: rejects a poll interval outside the budget + template: migration-job.yaml + set: {"deploymentOrdering.pollIntervalSeconds": 1200} + asserts: + - failedTemplate: + errorMessage: deploymentOrdering.timeoutSeconds must exceed pollIntervalSeconds + + - it: rejects an empty helper image + template: migration-job.yaml + set: {"deploymentOrdering.image": ""} + asserts: + - failedTemplate: + errorMessage: deploymentOrdering.image is required + + - it: does not validate unused disabled settings + template: migration-job.yaml + set: {"deploymentOrdering.enabled": false, "deploymentOrdering.image": "", "deploymentOrdering.timeoutSeconds": 0} + asserts: + - {"hasDocuments": {"count": 0}} + + - it: rejects a negative retry count + template: migration-job.yaml + set: {"deploymentOrdering.backoffLimit": -1} + asserts: + - failedTemplate: + errorMessage: deploymentOrdering.backoffLimit must be a nonnegative integer + + - it: preserves the content suffix with long release names + template: migration-job.yaml + documentIndex: 0 + release: {"name": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "revision": 101} + asserts: + - {"matchRegex": {"path": "metadata.name", "pattern": "^.{1,56}$"}} + - {"matchRegex": {"path": "metadata.name", "pattern": "-migrate-[a-f0-9]{16}$"}} + + - it: bounds immutable snapshot names with long release names + template: migration-job.yaml + documentIndex: 1 + release: {"name": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "revision": 101} + asserts: + - matchRegex: + path: metadata.name + pattern: '^.{1,63}$' + - matchRegex: + path: metadata.name + pattern: '-migrate-[a-f0-9]{16}-config$' + + - it: uses a bounded stable ordering name with long release names + template: api-deployment.yaml + release: {"name": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "revision": 101} + asserts: + - equal: + path: spec.template.spec.initContainers[0].env[1].name + value: ORDERING_CONFIGMAP + - matchRegex: + path: spec.template.spec.initContainers[0].env[1].value + pattern: '^.{1,58}$' + - matchRegex: + path: spec.template.spec.initContainers[0].env[1].value + pattern: '-[a-f0-9]{8}-ordering$' + + - it: gates a partially specified automation writer + template: brainstore-automationwriter-deployment.yaml + set: {"brainstore.automationwriter": {"replicas": 1}} + asserts: + - {"equal": {"path": "spec.template.spec.initContainers[0].name", "value": "wait-for-migrations"}} + - {"equal": {"path": "spec.replicas", "value": 1}} + + - it: reserves the desired pod fingerprint without discarding custom annotations + template: brainstore-reader-deployment.yaml + set: + brainstore.reader.annotations.pod: + braintrust.dev/pod-template: forged + custom: preserved + asserts: + - matchRegex: + path: spec.template.metadata.annotations["braintrust.dev/pod-template"] + pattern: "^[a-f0-9]{64}$" + - equal: + path: spec.template.metadata.annotations.custom + value: preserved + + - it: gives the API time for migration and one direct rollout check + template: api-deployment.yaml + set: + deploymentOrdering.timeoutSeconds: 900 + asserts: + - equal: + path: spec.progressDeadlineSeconds + value: 2100 + + - it: removes both Loop waits when ordering is disabled + template: loop-runtime-deployment.yaml + values: + - __fixtures__/loop-runtime-values.yaml + set: + deploymentOrdering.enabled: false + asserts: + - isNull: + path: spec.template.spec.initContainers + + - it: uses the shared API floor for Gateway startup + template: ai-gateway-deployment.yaml + set: + aiGateway.enabled: true + api.image.tag: v2.15.0 + asserts: + - contains: + path: spec.template.spec.initContainers[1].env + content: {name: BRAINSTORE_MINIMUM_VERSION, value: "2.15.0"} + + - it: applies the explicit shared floor to Loop startup + template: loop-runtime-deployment.yaml + values: + - __fixtures__/loop-runtime-values.yaml + set: + api.image.tag: deadbeef + deploymentOrdering.minimumVersion: v2.15.0 + deploymentOrdering.migrationImage: public.ecr.aws/braintrust/standalone-api:v2.23.0 + asserts: + - contains: + path: spec.template.spec.initContainers[1].env + content: {name: BRAINSTORE_MINIMUM_VERSION, value: "2.15.0"} diff --git a/braintrust/values.yaml b/braintrust/values.yaml index 3583756..1d3bbbe 100644 --- a/braintrust/values.yaml +++ b/braintrust/values.yaml @@ -102,26 +102,6 @@ btqlAuditLogsBestEffortOrgIds: [] api: name: "braintrust-api" - # Experimental: hold new API processes until compatible Brainstore capacity - # is available. Shared by all API pools. Enable for a canary before promotion. - # Requires stable release tags (vX.Y.Z); opaque custom tags cannot prove compatibility. - # Only the init container mounts a Kubernetes API token when enabled; custom - # sidecars needing Kubernetes API access must project their own token. - brainstoreStartupGate: - enabled: true - # Empty derives the floor from each pool's api.image.tag. Kept in each API Pod so - # older APIs can restart against newer Brainstore releases in the same major. - minimumVersion: "" - image: "python:3.13-alpine@sha256:79e7a9b9ff1cbceff819f856fb374477792a5967759d94df266de7b7b4120e6f" - timeoutSeconds: 1200 - pollIntervalSeconds: 15 - resources: - requests: - cpu: "25m" - memory: "32Mi" - limits: - cpu: "100m" - memory: "128Mi" labels: {} podLabels: {} annotations: @@ -407,6 +387,7 @@ aiGateway: deployment: {} service: {} pod: {} + serviceaccount: {} # With autoscaling, the default minReplicas. replicas: 2 # Needs metrics-server; without it the HPA holds minReplicas. @@ -429,6 +410,8 @@ aiGateway: type: ClusterIP port: 8080 portName: http + serviceAccount: + name: "braintrust-ai-gateway" # Optional cloud region identifier (e.g. us-east-1). Sets GATEWAY_REGION when provided. region: "" braintrustAppUrl: "https://www.braintrust.dev" @@ -831,3 +814,29 @@ virtualService: host: "braintrust-api" # Should match api.service.name or api.name port: number: 8000 # Should match api.service.port + +# Order startup as migrations -> completed Brainstore rollout -> all other apps. +# Disable to deploy everything without chart-managed startup ordering. +# Unchanged workloads keep running; successful migration inputs are reused. +# See README.md#release-ordering for configuration and operational limits. +deploymentOrdering: + enabled: true + # Empty derives the API compatibility floor from each pool's stable image tag. + # Required for opaque API/Brainstore tags; asserts the opaque Brainstore release. + minimumVersion: "" + # Full standalone-API image containing migrations for all selected service images. + # Empty selects the newest stable API/Brainstore release; required for opaque pins. + migrationImage: "" + # Separate budget for each phase (migration Job/wait, then Brainstore rollout). + timeoutSeconds: 1200 + pollIntervalSeconds: 15 + backoffLimit: 2 + # Small read-only init-container helper; separate from the migration bundle above. + image: "python:3.13-alpine@sha256:79e7a9b9ff1cbceff819f856fb374477792a5967759d94df266de7b7b4120e6f" + resources: + requests: + cpu: 25m + memory: 32Mi + limits: + cpu: 100m + memory: 128Mi diff --git a/tests/test_brainstore_startup_gate.py b/tests/test_brainstore_startup_gate.py index 3575496..cb7f272 100644 --- a/tests/test_brainstore_startup_gate.py +++ b/tests/test_brainstore_startup_gate.py @@ -1,5 +1,6 @@ import importlib.util import io +import json import os from pathlib import Path import tempfile @@ -14,95 +15,84 @@ spec.loader.exec_module(gate) -def pod(version="v2.15.0", uid="new", is_ready=True, **metadata): +TEMPLATE = "a" * 64 +OLD_TEMPLATE = "b" * 64 +LATER_TEMPLATE = "c" * 64 + + +def target(template=TEMPLATE, replicas=1, min_ready=0, name="reader", + container="brainstore-reader"): + return {"deployment": name, "container": container, "template": template, + "replicas": replicas, "minReadySeconds": min_ready} + + +def ordering(*targets, job_name="release-migrate-" + "d" * 32): + return {"data": {"ordering.json": json.dumps({ + "job": job_name, "targets": list(targets) if targets else [target()], + })}} + + +def pod(version="v2.15.0", uid="new", is_ready=True, template=TEMPLATE, + container="brainstore-reader", image=None, declared_version=None, **metadata): + image = image if image is not None else "registry/brainstore:" + version + if template is not None: + metadata.setdefault("annotations", {})["braintrust.dev/pod-template"] = template + if declared_version is not None: + metadata.setdefault("annotations", {})["braintrust.dev/brainstore-version"] = declared_version return { "metadata": {"name": uid, "uid": uid, **metadata}, - "spec": {"containers": [{"name": "brainstore-reader", "image": "registry/brainstore:" + version}]}, + "spec": {"containers": [{"name": container, "image": image}]}, "status": { "phase": "Running", "conditions": [{"type": "Ready", "status": "True" if is_ready else "False"}], - "containerStatuses": [{"name": "brainstore-reader", "image": "registry/brainstore:" + version, + "containerStatuses": [{"name": container, "image": image, "ready": is_ready, "state": {"running": {}}}], }, } -def deployment(version="v2.15.0", replicas=1): +def deployment(version="v2.15.0", replicas=1, template=TEMPLATE, container="brainstore-reader", + image=None, declared_version=None, min_ready=0): return { "metadata": {"name": "reader", "uid": "reader-deployment", "generation": 2}, - "spec": {"replicas": replicas, "template": {"spec": pod(version)["spec"]}}, - "status": {"observedGeneration": 2, "availableReplicas": replicas}, + "spec": {"replicas": replicas, "minReadySeconds": min_ready, "template": { + "metadata": {"annotations": pod( + template=template, declared_version=declared_version, + )["metadata"].get("annotations", {})}, + "spec": pod(version, container=container, image=image)["spec"], + }}, + "status": {"observedGeneration": 2, "updatedReplicas": replicas, + "readyReplicas": replicas, "availableReplicas": replicas, "replicas": replicas}, } class CompatibilityTests(unittest.TestCase): def check(self, resource=None, pods=None, minimum="2.15.0"): - return gate.check_deployment(resource or deployment(), pods if pods is not None else [pod()], - "brainstore-reader", gate.release_version(minimum)) + return gate.check_deployment( + resource if resource is not None else deployment(), + pods if pods is not None else [pod()], + target(), gate.release_version(minimum), + ) def test_new_api_waits_when_helm_has_not_updated_brainstore_spec(self): - self.assertIn("desired image", self.check(deployment("v2.14.0"), [pod("v2.14.0")])) + self.assertIsNotNone(self.check(deployment("v2.14.0"))) def test_new_api_waits_for_incompatible_old_pods(self): - self.assertIn("incompatible Pod old", self.check(pods=[pod(), pod("v2.14.0", "old")])) + self.assertIsNotNone(self.check(pods=[pod(), pod("v2.14.0", "old")])) def test_incompatible_terminating_pod_blocks_even_when_unready(self): old = pod("v2.14.0", "old", False, deletionTimestamp="2026-09-30T12:00:00Z") - self.assertIn("incompatible Pod old", self.check(pods=[pod(), old])) - - def test_terminal_pods_do_not_block(self): - for phase in ("Succeeded", "Failed"): - with self.subTest(phase=phase): - old = pod("v2.14.0", "old", False) - old["status"]["phase"] = phase - self.assertIsNone(self.check(pods=[pod(), old])) - - def test_matching_ready_fleet_passes(self): - self.assertIsNone(self.check()) - - def test_requires_live_ready_capacity(self): - self.assertIn("available compatible", self.check(pods=[pod(is_ready=False)])) - self.assertIn("available compatible", self.check(deployment(replicas=2), [pod()])) - - def test_honors_deployment_min_ready_seconds_availability(self): - resource = deployment() - resource["status"]["availableReplicas"] = 0 - self.assertIn("available compatible", self.check(resource)) + self.assertIsNotNone(self.check(pods=[pod(), old])) def test_old_api_can_restart_against_newer_brainstore(self): self.assertIsNone(self.check(deployment("v2.16.0"), [pod("v2.16.0")], "2.15.0")) - def test_old_api_can_restart_during_compatible_future_rollout(self): - resource = deployment("v2.16.0") - pods = [pod("v2.15.0", "old"), pod("v2.16.0", "new", False)] - self.assertIsNone(self.check(resource, pods)) - - def test_new_api_still_blocks_during_same_future_rollout(self): - pods = [pod("v2.15.0", "old"), pod("v2.16.0", "new", False)] - self.assertIn("incompatible", self.check(deployment("v2.16.0"), pods, "2.16.0")) - - def test_compatible_terminating_pod_does_not_block_restart(self): - terminating = pod("v2.15.0", "old", False, deletionTimestamp="2026-09-30T12:00:00Z") - self.assertIsNone(self.check(pods=[pod(), terminating])) - - def test_zero_replica_role_waits_for_remaining_pods(self): - self.assertIsNone(self.check(deployment(replicas=0), [])) - self.assertIn("scaled-down", self.check(deployment(replicas=0), [pod()])) - - def test_unobserved_generation_does_not_pass(self): - resource = deployment() - resource["status"]["observedGeneration"] = 1 - self.assertIn("not observed", self.check(resource)) - - def test_deleted_deployment_does_not_pass(self): - resource = deployment() - resource["metadata"]["deletionTimestamp"] = "2026-09-30T12:00:00Z" - self.assertIn("being deleted", self.check(resource)) - def test_unknown_and_other_major_images_fail_closed(self): - for version in ("latest", "deadbeef", "v3.0.0", "v2.15.0-rc.1"): + for version in ("latest", "deadbeef", "0123456", "0" * 40, + "v3.0.0", "v2.15.0-rc.1"): with self.subTest(version=version): - self.assertIn("incompatible", self.check(pods=[pod(version)])) + self.assertIsNotNone(self.check(deployment(version))) + self.assertIsNotNone(self.check(pods=[pod(version)])) def test_registry_port_and_tagged_digest_are_supported(self): image = "localhost:5000/brainstore:v2.15.0@sha256:abc" @@ -122,7 +112,10 @@ def test_stable_release_parser(self): def test_missing_expected_container_fails_closed(self): wrong = pod() wrong["spec"]["containers"][0]["name"] = "sidecar" - self.assertIn("incompatible", self.check(pods=[wrong])) + self.assertIsNotNone(self.check(pods=[wrong])) + resource = deployment() + resource["spec"]["template"]["spec"]["containers"][0]["name"] = "sidecar" + self.assertIsNotNone(self.check(resource)) def test_runtime_image_aliases_do_not_block_a_compatible_deployment_pod(self): for image in ("registry/brainstore:deadbeef", "registry/brainstore@sha256:abc"): @@ -134,12 +127,450 @@ def test_runtime_image_aliases_do_not_block_a_compatible_deployment_pod(self): def test_runtime_alias_does_not_override_an_incompatible_pod_spec(self): current = pod("v2.14.0") current["status"]["containerStatuses"][0]["image"] = "registry/brainstore:v2.15.0" - self.assertIn("incompatible", self.check(pods=[current])) + self.assertIsNotNone(self.check(pods=[current])) + + +class OpaqueImageTests(unittest.TestCase): + def rollout(self, image="registry/brainstore:deadbeef", declared_version="2.15.0"): + return ( + deployment(image=image, declared_version=declared_version), + pod(image=image, declared_version=declared_version), + ) + + def check(self, resource, pods, minimum="2.15.0"): + return gate.check_deployment( + resource, pods, target(replicas=resource["spec"]["replicas"]), + gate.release_version(minimum), + ) + + def test_sha_custom_prerelease_and_digest_pins_accept_explicit_assertions(self): + for image in ( + "registry/brainstore:deadbeef", "registry/brainstore:" + "a" * 40, + "registry/brainstore:0123456", "registry/brainstore:" + "0" * 40, + "registry/brainstore:custom-build", "registry/brainstore:latest", + "registry/brainstore:v2.15.0-rc.1", + "localhost:5000/brainstore@sha256:abc", + "registry/brainstore:deadbeef@sha256:abc", + ): + for declared_version in ("2.15.0", "v2.16.0"): + with self.subTest(image=image, declared_version=declared_version): + resource, current = self.rollout(image, declared_version) + self.assertIsNone(self.check(resource, [current])) + + def test_missing_or_invalid_template_assertion_blocks_matching_pods(self): + for declared_version in (None, "", "unknown", "2.015.0", "2.15", + "2.15.0-rc.1", "2.15.0+build", "2.15.0\n", 215, True): + with self.subTest(declared_version=declared_version): + resource, current = self.rollout(declared_version=declared_version) + self.assertIsNotNone(self.check(resource, [current])) + + def test_lower_or_other_major_template_assertions_cannot_be_overridden_by_pods(self): + for declared_version in ("2.14.99", "1.99.0", "3.0.0"): + with self.subTest(declared_version=declared_version): + resource, current = self.rollout(declared_version=declared_version) + self.assertIsNotNone(self.check(resource, [current])) + current["metadata"]["annotations"]["braintrust.dev/brainstore-version"] = "2.15.0" + self.assertIsNotNone(self.check(resource, [current])) + + def test_every_live_pod_must_have_the_exact_template_assertion(self): + for declared_version in (None, "", "unknown", "2.14.0", "3.0.0", + "2.16.0", "v2.15.0", 215): + with self.subTest(declared_version=declared_version): + resource, _ = self.rollout() + current = pod(image="registry/brainstore:deadbeef", + declared_version=declared_version) + self.assertIsNotNone(self.check(resource, [current])) + resource["spec"]["replicas"] = 2 + for counter in ("updatedReplicas", "readyReplicas", "availableReplicas", "replicas"): + resource["status"][counter] = 2 + stray = pod(uid="stray", image="registry/brainstore:deadbeef", + declared_version="2.15.0") + self.assertIsNotNone(self.check(resource, [stray, current])) + + def test_matching_assertion_does_not_allow_a_different_image_or_stray_backend(self): + for desired_image, other_image in ( + ("registry/brainstore:deadbeef", "registry/brainstore:feedface"), + ("registry/brainstore:deadbeef", "other/brainstore:deadbeef"), + ("registry/brainstore:deadbeef", "registry/brainstore:v2.15.0"), + ("registry/brainstore:deadbeef@sha256:abc", "registry/brainstore:deadbeef"), + ("registry/brainstore@sha256:abc", "registry/brainstore@sha256:def"), + ): + with self.subTest(desired_image=desired_image, other_image=other_image): + resource, _ = self.rollout(desired_image) + stray = pod(uid="stray", image=other_image, declared_version="2.15.0") + self.assertIsNotNone(self.check(resource, [stray])) + resource["spec"]["replicas"] = 2 + for counter in ("updatedReplicas", "readyReplicas", "availableReplicas", "replicas"): + resource["status"][counter] = 2 + current = pod(image=desired_image, declared_version="2.15.0") + self.assertIsNotNone(self.check(resource, [current, stray])) + + def test_assertion_does_not_replace_a_missing_desired_or_pod_container(self): + resource, current = self.rollout() + current["spec"]["containers"] = [] + self.assertIsNotNone(self.check(resource, [current])) + resource, current = self.rollout() + resource["spec"]["template"]["spec"]["containers"] = [] + self.assertIsNotNone(self.check(resource, [current])) + + def test_forged_assertions_cannot_override_stable_image_versions(self): + for image in ("registry/brainstore:v2.14.0", "registry/brainstore:v3.0.0", + "registry/brainstore:v2.14.0@sha256:abc", + "registry/brainstore:v3.0.0@sha256:abc"): + with self.subTest(image=image): + resource, current = self.rollout(image) + self.assertIsNotNone(self.check(resource, [current])) + compatible_pod = pod(declared_version="2.15.0") + self.assertIsNotNone(self.check(resource, [compatible_pod])) + resource = deployment(declared_version="2.15.0") + self.assertIsNotNone(self.check(resource, [current])) + + def test_stable_images_use_actual_versions_not_assertions(self): + for declared_version in (None, "invalid", "1.0.0", "3.0.0"): + with self.subTest(declared_version=declared_version): + resource, current = self.rollout("registry/brainstore:v2.16.0", + declared_version) + self.assertIsNone(self.check(resource, [current])) + resource = deployment() + _, opaque = self.rollout() + self.assertIsNotNone(self.check(resource, [opaque])) + + def test_current_assertion_still_requires_a_completed_observed_rollout(self): + for change in ("generation", "template", "pod_template", "terminating", + "unready", "missing_pod", "surge", + "updatedReplicas", "readyReplicas", "availableReplicas", "replicas"): + with self.subTest(change=change): + resource, current = self.rollout() + pods = [current] + if change == "generation": + resource["status"]["observedGeneration"] = 1 + elif change == "template": + resource["spec"]["template"]["metadata"]["annotations"][ + "braintrust.dev/pod-template"] = OLD_TEMPLATE + elif change == "pod_template": + current["metadata"]["annotations"]["braintrust.dev/pod-template"] = OLD_TEMPLATE + elif change == "terminating": + current["metadata"]["deletionTimestamp"] = "2026-09-30T12:00:00Z" + elif change == "unready": + current["status"]["containerStatuses"][0]["ready"] = False + elif change == "missing_pod": + pods = [] + elif change == "surge": + pods.append(pod(uid="surge", image="registry/brainstore:deadbeef", + declared_version="2.15.0")) + else: + resource["status"][change] = 0 + self.assertIsNotNone(self.check(resource, pods)) + + def test_older_api_uses_later_opaque_rollouts_own_assertion(self): + resource, current = self.rollout("registry/brainstore:later-sha", "2.16.0") + self.assertIsNone(self.check(resource, [current])) + self.assertIsNotNone(self.check(resource, [current], minimum="2.17.0")) + current["metadata"]["annotations"]["braintrust.dev/brainstore-version"] = "2.15.0" + self.assertIsNotNone(self.check(resource, [current])) + resource, current = self.rollout("registry/brainstore:major-three", "3.0.0") + self.assertIsNotNone(self.check(resource, [current])) + self.assertIsNone(self.check(resource, [current], minimum="3.0.0")) + + + def test_later_opaque_major_rollout_resets_an_older_apis_confirmation(self): + first, first_pod = self.rollout() + later, later_pod = self.rollout("registry/brainstore:major-three", "3.0.0") + client = api_client() + client.deployment.side_effect = [first, later, later] + client.pods.side_effect = [[first_pod], [later_pod], [later_pod]] + clock = FakeClock() + with self.assertRaisesRegex(TimeoutError, "declared Brainstore version"): + gate.wait_for_fleet( + client, "release-ordering", (2, 15, 0), 45, 15, + clock.clock, clock.sleep, Mock(), + ) + self.assertEqual(client.deployment.call_count, 3) - def test_pod_ready_without_expected_running_container_is_insufficient(self): + def test_opaque_zero_role_needs_drain_not_a_compatible_assertion(self): + for declared_version in (None, "invalid", "1.0.0", "3.0.0"): + with self.subTest(declared_version=declared_version): + resource = deployment(replicas=0, image="registry/brainstore@sha256:abc", + declared_version=declared_version) + self.assertIsNone(self.check(resource, [])) + current = pod(image="registry/brainstore@sha256:abc", + declared_version=declared_version) + self.assertIsNotNone(self.check(resource, [current])) + resource["status"]["replicas"] = 1 + self.assertIsNotNone(self.check(resource, [])) + + +def migration_job(*conditions): + return {"status": {"conditions": [ + {"type": condition, "status": "True"} for condition in conditions + ]}} + + +def api_client(configs=None, jobs=None): + configs = iter(configs) if configs is not None else None + jobs = iter(jobs) if jobs is not None else None + + def get(path, deadline): + if "/configmaps/" in path: + resource = next(configs) if configs is not None else ordering() + elif "/jobs/" in path: + resource = next(jobs) if jobs is not None else migration_job("Complete") + else: + raise AssertionError(f"unexpected API path: {path}") + if isinstance(resource, BaseException): + raise resource + return resource + + client = Mock(namespace="ns") + client.get.side_effect = get + return client + + +class RolloutTests(unittest.TestCase): + def check(self, resource=None, pods=None, expected=None): + return gate.check_deployment( + resource if resource is not None else deployment(), + pods if pods is not None else [pod()], + expected if expected is not None else target(), (2, 15, 0), + ) + + def test_same_image_content_change_waits_for_current_pods(self): + old = pod(template=OLD_TEMPLATE) + self.assertIsNotNone(self.check(pods=[old, pod(is_ready=False)])) + self.assertIsNotNone(self.check(pods=[old])) + self.assertIsNone(self.check(pods=[pod()])) + + def test_env_change_waits_for_updated_replicas(self): + resource = deployment() + resource["metadata"]["generation"] = 3 + resource["status"]["observedGeneration"] = 3 + resource["spec"]["template"]["spec"]["containers"][0]["env"] = [ + {"name": "BRAINSTORE_CONFIG", "value": "new-config"}, + ] + resource["status"]["updatedReplicas"] = 0 + self.assertIsNotNone(self.check(resource)) + resource["status"]["updatedReplicas"] = 1 current = pod() - current["status"]["containerStatuses"] = [] - self.assertIn("available compatible", self.check(pods=[current])) + current["spec"] = resource["spec"]["template"]["spec"] + self.assertIsNone(self.check(resource, [current])) + + def test_old_complete_generation_cannot_pass_before_helm_updates_content(self): + self.assertIsNotNone(self.check(deployment(template=OLD_TEMPLATE), + [pod(template=OLD_TEMPLATE)])) + + def test_exact_expected_content_is_required_on_deployment_and_live_pods(self): + for template in (None, "", "invalid", "a" * 63, OLD_TEMPLATE, LATER_TEMPLATE): + with self.subTest(template=template): + self.assertIsNotNone(self.check(deployment(template=template))) + self.assertIsNotNone(self.check(pods=[pod(template=template)])) + self.assertIsNone(self.check(deployment(template=LATER_TEMPLATE), + [pod(template=LATER_TEMPLATE)], + target(template=LATER_TEMPLATE))) + + def test_expected_replica_count_blocks_stale_healthy_deployment(self): + for replicas in (0, 2): + with self.subTest(replicas=replicas): + current = deployment(replicas=replicas) + pods = [pod(uid=str(index)) for index in range(replicas)] + self.assertIsNotNone(self.check(current, pods)) + self.assertIsNone(self.check(current, pods, target(replicas=replicas))) + self.assertIsNotNone(self.check(expected=target(replicas=0))) + + def test_expected_min_ready_seconds_blocks_stale_healthy_deployment(self): + self.assertIsNotNone(self.check(expected=target(min_ready=60))) + self.assertIsNone(self.check(deployment(min_ready=60), [pod()], target(min_ready=60))) + self.assertIsNotNone(self.check(deployment(min_ready=60))) + + def test_malformed_deployment_counts_fail_closed(self): + for field in ("replicas", "minReadySeconds"): + for value in (True, False, None, -1, "1", 1.0): + with self.subTest(field=field, value=value): + resource = deployment() + resource["spec"][field] = value + self.assertIsNotNone(self.check(resource)) + + def test_unobserved_or_invalid_generation_does_not_pass(self): + for generation in (None, 0, -1, True, "2", 2.0): + with self.subTest(generation=generation): + resource = deployment() + resource["metadata"]["generation"] = generation + self.assertIsNotNone(self.check(resource)) + for observed in (None, 0, 1, -1, True, "2", 2.0): + with self.subTest(observed=observed): + resource = deployment() + resource["status"]["observedGeneration"] = observed + self.assertIsNotNone(self.check(resource)) + + def test_deleted_deployment_does_not_pass(self): + resource = deployment() + resource["metadata"]["deletionTimestamp"] = "2026-09-30T12:00:00Z" + self.assertIsNotNone(self.check(resource)) + + def test_every_replica_counter_must_equal_expected_count(self): + for counter in ("updatedReplicas", "readyReplicas", "availableReplicas", "replicas"): + for count in (None, 0, 2, True, False, "1", 1.0, -1): + with self.subTest(counter=counter, count=count): + resource = deployment() + if count is None: + del resource["status"][counter] + else: + resource["status"][counter] = count + self.assertIsNotNone(self.check(resource)) + + def test_requires_exactly_desired_live_ready_current_pods(self): + for pods in ([], [pod(is_ready=False)], [pod(), pod(uid="surge")], + [pod(deletionTimestamp="2026-09-30T12:00:00Z")], + [pod(), pod(uid="terminating", is_ready=False, + deletionTimestamp="2026-09-30T12:00:00Z")]): + with self.subTest(pods=pods): + self.assertIsNotNone(self.check(pods=pods)) + for change in ("missing", "not_running", "not_ready", "pod_not_ready", "pending"): + with self.subTest(change=change): + current = pod() + status = current["status"] + if change == "missing": + status["containerStatuses"] = [] + elif change == "not_running": + status["containerStatuses"][0]["state"] = {"waiting": {}} + elif change == "not_ready": + status["containerStatuses"][0]["ready"] = False + elif change == "pod_not_ready": + status["conditions"] = [] + else: + status["phase"] = "Pending" + self.assertIsNotNone(self.check(pods=[current])) + + def test_stray_service_backend_also_requires_expected_content(self): + stray = pod(uid="stray", template=OLD_TEMPLATE) + self.assertIsNotNone(self.check(pods=[stray])) + del stray["metadata"]["annotations"] + self.assertIsNotNone(self.check(pods=[stray])) + + def test_terminal_pods_do_not_block(self): + for phase in ("Succeeded", "Failed"): + with self.subTest(phase=phase): + old = pod("v2.14.0", template=OLD_TEMPLATE) + old["status"]["phase"] = phase + self.assertIsNone(self.check(pods=[pod(), old])) + + def test_honors_deployment_min_ready_seconds_availability(self): + resource = deployment(min_ready=60) + resource["status"]["availableReplicas"] = 0 + self.assertIsNotNone(self.check(resource, expected=target(min_ready=60))) + resource["status"]["availableReplicas"] = 1 + self.assertIsNone(self.check(resource, expected=target(min_ready=60))) + + def test_zero_role_requires_expected_content_and_completed_scale_down(self): + expected = target(replicas=0) + resource = deployment(replicas=0) + self.assertIsNone(self.check(resource, [], expected)) + self.assertIsNotNone(self.check(resource, [pod()], expected)) + for counter in ("updatedReplicas", "readyReplicas", "availableReplicas", "replicas"): + for count in (1, -1, True, False, "0", 0.0): + with self.subTest(counter=counter, count=count): + resource["status"][counter] = count + self.assertIsNotNone(self.check(resource, [], expected)) + del resource["status"][counter] + self.assertIsNone(self.check(resource, [], expected)) + for template in (None, OLD_TEMPLATE): + with self.subTest(template=template): + self.assertIsNotNone(self.check( + deployment(replicas=0, template=template), [], expected, + )) + resource["status"]["observedGeneration"] = 1 + self.assertIsNotNone(self.check(resource, [], expected)) + + def test_zero_role_does_not_require_an_inactive_image_or_container(self): + expected = target(replicas=0) + for version in ("v2.14.0", "v3.0.0", "deadbeef", "v2.15.0-rc.1"): + with self.subTest(version=version): + self.assertIsNone(self.check(deployment(version, replicas=0), [], expected)) + resource = deployment(replicas=0) + resource["spec"]["template"]["spec"]["containers"] = [] + self.assertIsNone(self.check(resource, [], expected)) + terminating = pod(is_ready=False, deletionTimestamp="2026-09-30T12:00:00Z") + self.assertIsNotNone(self.check(resource, [terminating], expected)) + for phase in ("Succeeded", "Failed"): + with self.subTest(phase=phase): + terminal = pod("v2.14.0", template=OLD_TEMPLATE) + terminal["status"]["phase"] = phase + self.assertIsNone(self.check(resource, [terminal], expected)) + + def test_older_application_accepts_latest_expected_compatible_rollout_only(self): + resource = deployment("v2.16.0", template=LATER_TEMPLATE) + expected = target(template=LATER_TEMPLATE) + later = pod("v2.16.0", template=LATER_TEMPLATE) + self.assertIsNone(self.check(resource, [later], expected)) + self.assertIsNotNone(self.check(resource, [pod()], expected)) + resource["status"]["updatedReplicas"] = 0 + self.assertIsNotNone(self.check(resource, [later], expected)) + + +class OrderingConfigTests(unittest.TestCase): + def load(self, resource): + client = Mock(namespace="custom%20namespace") + client.get.return_value = resource + return gate.load_ordering_config(client, "release-ordering", 100) + + def test_loads_latest_config_from_api_and_preserves_zero_targets(self): + client = Mock(namespace="custom%20namespace") + client.get.side_effect = [ordering(), ordering(target(replicas=0, min_ready=60))] + first = gate.load_ordering_config(client, "release/ordering", 100) + second = gate.load_ordering_config(client, "release/ordering", 100) + self.assertEqual(first["targets"], [target()]) + self.assertEqual(second["targets"], [target(replicas=0, min_ready=60)]) + self.assertEqual([call.args for call in client.get.call_args_list], [ + ("/api/v1/namespaces/custom%20namespace/configmaps/release%2Fordering", 100), + ] * 2) + + def test_missing_or_malformed_configmap_data_fails_closed(self): + for resource in (None, [], {}, {"data": None}, {"data": []}, {"data": {}}, + {"data": {"ordering.json": None}}, + {"data": {"ordering.json": ""}}, + {"data": {"ordering.json": "not JSON"}}): + with self.subTest(resource=resource), self.assertRaises((KeyError, TypeError, ValueError)): + self.load(resource) + + def test_job_and_nonempty_target_list_are_required(self): + for config in (None, [], {}, {"job": "job"}, {"targets": [target()]}, + {"job": "job", "targets": []}, + {"job": "job", "targets": {}}, + {"job": "job", "targets": [None]}, + {"job": "job", "targets": [{}]}): + with self.subTest(config=config), self.assertRaises(ValueError): + self.load({"data": {"ordering.json": json.dumps(config)}}) + for job_name in (None, "", True, 123, [], {}): + with self.subTest(job_name=job_name), self.assertRaises(ValueError): + self.load(ordering(job_name=job_name)) + + def test_every_target_requires_names_and_sha256_content(self): + invalid = { + "deployment": (None, "", True, 1, [], {}), + "container": (None, "", True, 1, [], {}), + "template": (None, "", True, 1, "a" * 63, "a" * 65, "A" * 64, "z" * 64, + TEMPLATE + "\n"), + } + for field, values in invalid.items(): + for value in values: + with self.subTest(field=field, value=value), self.assertRaises(ValueError): + current = target() + current[field] = value + self.load(ordering(target(), current)) + for field in target(): + with self.subTest(missing=field), self.assertRaises(ValueError): + current = target() + del current[field] + self.load(ordering(current)) + + def test_replica_and_min_ready_shapes_are_nonnegative_integers_not_booleans(self): + for field in ("replicas", "minReadySeconds"): + for value in (None, -1, True, False, 1.0, "1", [], {}): + with self.subTest(field=field, value=value), self.assertRaises(ValueError): + current = target() + current[field] = value + self.load(ordering(current)) + self.assertEqual(self.load(ordering(target(replicas=0, min_ready=0)))["targets"], + [target(replicas=0, min_ready=0)]) class FakeClock: @@ -154,86 +585,246 @@ def sleep(self, seconds): class WaitTests(unittest.TestCase): - def wait(self, observations, timeout=60): + def wait(self, observations, timeout=60, configs=None, jobs=None): clock = FakeClock() log = Mock() + client = api_client(configs, jobs) with patch.object(gate, "check_fleet", side_effect=observations) as check: - gate.wait_for_fleet(Mock(), [{"deployment": "reader", "container": "brainstore-reader"}], - (2, 15, 0), timeout, 15, clock.clock, clock.sleep, log) - return clock, log, check + gate.wait_for_fleet(client, "release-ordering", (2, 15, 0), timeout, 15, + clock.clock, clock.sleep, log) + return clock, log, check, client - def test_requires_two_successful_observations(self): - clock, _, check = self.wait([None, None]) + def test_requires_two_successful_observations_and_reads_config_and_job_each_time(self): + clock, _, check, client = self.wait([None, None]) self.assertEqual(check.call_count, 2) + self.assertEqual(client.get.call_count, 4) self.assertEqual(clock.now, 15) def test_failed_check_resets_confirmation(self): - _, _, check = self.wait([None, "old pod remains", None, None]) + _, _, check, _ = self.wait([None, "old pod remains", None, None]) self.assertEqual(check.call_count, 4) + def test_every_expected_state_change_resets_confirmation(self): + for changed in (ordering(target(template=LATER_TEMPLATE)), ordering(target(replicas=2)), + ordering(target(min_ready=60)), ordering(target(name="new-reader")), + ordering(job_name="new-job")): + with self.subTest(changed=changed): + clock, _, check, _ = self.wait([None, None, None], + configs=[ordering(), changed, changed]) + self.assertEqual(check.call_count, 3) + self.assertEqual(clock.now, 30) + def test_transient_missing_deployment_and_rbac_are_retried(self): - errors = [urllib.error.HTTPError("https://api", code, "synthetic", None, None) + errors = [urllib.error.HTTPError("https://api", code, "synthetic", {}, io.BytesIO()) for code in (404, 403)] - _, _, check = self.wait(errors + [None, None]) + _, _, check, _ = self.wait(errors + [None, None]) self.assertEqual(check.call_count, 4) + def test_failed_api_reads_reset_successful_observations(self): + for error in (OSError("synthetic"), ValueError("synthetic"), + KeyError("synthetic"), TypeError("synthetic")): + with self.subTest(error=type(error).__name__): + clock, _, check, _ = self.wait([None, error, None, None]) + self.assertEqual(check.call_count, 4) + self.assertEqual(clock.now, 45) + + def test_invalid_config_resets_confirmation_without_observing_deployments(self): + clock, _, check, client = self.wait( + [None, None, None], configs=[ordering(), {"data": {}}, ordering(), ordering()], + ) + self.assertEqual(client.get.call_count, 7) + self.assertEqual(check.call_count, 3) + self.assertEqual(clock.now, 45) + + def test_current_migration_job_must_be_complete_on_every_observation(self): + clock, _, check, client = self.wait( + [None, None, None], timeout=75, + configs=[ordering(), ordering(job_name="new-job"), + ordering(job_name="new-job"), ordering(job_name="new-job")], + jobs=[migration_job("Complete"), migration_job(), + migration_job("Complete"), migration_job("Complete")], + ) + self.assertEqual(check.call_count, 3) + self.assertEqual(clock.now, 45) + job_paths = [call.args[0] for call in client.get.call_args_list if "/jobs/" in call.args[0]] + self.assertEqual(job_paths[1:], ["/apis/batch/v1/namespaces/ns/jobs/new-job"] * 3) + + def test_pending_missing_or_failed_migration_never_releases_api(self): + for observation in (migration_job(), + urllib.error.HTTPError("https://api/jobs/job", 404, "", {}, io.BytesIO())): + with self.subTest(observation=observation), self.assertRaises(TimeoutError): + self.wait([], timeout=30, jobs=[observation, observation]) + with self.assertRaises(RuntimeError): + self.wait([], jobs=[migration_job("Failed", "Complete")]) + def test_timeout_never_releases_api(self): with self.assertRaisesRegex(TimeoutError, "old pod remains"): self.wait(["old pod remains"] * 3, timeout=30) def test_kubernetes_api_failure_never_releases_api(self): with self.assertRaisesRegex(TimeoutError, "HTTP 429"): - self.wait([urllib.error.HTTPError("https://api", 429, "synthetic", None, None)] * 3, 30) + self.wait([urllib.error.HTTPError("https://api", 429, "synthetic", {}, io.BytesIO())] * 3, 30) + + def test_missing_or_malformed_ordering_config_never_releases_api(self): + for resource in ({}, ordering(target(template="invalid")), ordering(job_name=""), + urllib.error.HTTPError("https://api/configmaps/gate", 404, "", {}, io.BytesIO())): + with self.subTest(resource=resource), self.assertRaises(TimeoutError): + self.wait([], timeout=30, configs=[resource, resource]) class FleetTests(unittest.TestCase): - def test_compatible_deployment_and_pod_churn_do_not_delay_an_old_api(self): - client = Mock() + def test_latest_compatible_content_requires_two_matching_observations(self): + client = api_client([ordering(), ordering(target(template=LATER_TEMPLATE)), + ordering(target(template=LATER_TEMPLATE))]) first = deployment("v2.15.0") - later = deployment("v2.16.0") + later = deployment("v2.16.0", template=LATER_TEMPLATE) later["metadata"]["generation"] = 3 later["status"]["observedGeneration"] = 3 - client.deployment.side_effect = [first, later] - client.pods.side_effect = [[pod("v2.15.0", "old")], [pod("v2.16.0", "new")]] - targets = [{"deployment": "reader", "container": "brainstore-reader"}] + client.deployment.side_effect = [first, later, later] + client.pods.side_effect = [[pod("v2.15.0", "old")], + [pod("v2.16.0", "new", template=LATER_TEMPLATE)], + [pod("v2.16.0", "new", template=LATER_TEMPLATE)]] clock = FakeClock() - gate.wait_for_fleet(client, targets, (2, 15, 0), 60, 15, + gate.wait_for_fleet(client, "release-ordering", (2, 15, 0), 60, 15, clock.clock, clock.sleep, Mock()) - self.assertEqual(client.deployment.call_count, 2) - self.assertEqual(clock.now, 15) + self.assertEqual(client.deployment.call_count, 3) + self.assertEqual(clock.now, 30) - def test_checks_all_three_roles_and_includes_stray_service_backends(self): + def test_checks_all_roles_and_includes_stray_service_backends(self): client = Mock() targets = [] deployments = [] pod_sets = [] - for role in ("reader", "writer", "fastreader"): + for role in ("reader", "writer", "fastreader", "automationwriter"): name = "custom-" + role container = "brainstore-" + role - targets.append({"deployment": name, "container": container}) - resource = deployment() + targets.append(target(name=name, container=container)) + resource = deployment(container=container) resource["metadata"]["name"] = name resource["metadata"]["uid"] = name - resource["spec"]["template"]["spec"]["containers"][0]["name"] = container - current = pod(uid=role) - current["spec"]["containers"][0]["name"] = container - current["status"]["containerStatuses"][0]["name"] = container deployments.append(resource) - pod_sets.append([current]) + pod_sets.append([pod(uid=role, container=container)]) client.deployment.side_effect = deployments client.pods.side_effect = pod_sets - reason = gate.check_fleet(client, targets, (2, 15, 0), 100) - self.assertIsNone(reason) - self.assertEqual(client.deployment.call_count, 3) + self.assertIsNone(gate.check_fleet(client, targets, (2, 15, 0), 100)) + self.assertEqual(client.deployment.call_count, 4) self.assertEqual([call.args[0] for call in client.pods.call_args_list], - ["app=custom-reader", "app=custom-writer", "app=custom-fastreader"]) - # A manually labelled Pod can receive Service traffic without belonging - # to a ReplicaSet, so it must not be filtered out by owner references. - stray = pod("v2.14.0", "stray") + ["app=custom-reader", "app=custom-writer", + "app=custom-fastreader", "app=custom-automationwriter"]) + self.assertTrue(all(call.args[1] == 100 for call in client.deployment.call_args_list)) + self.assertTrue(all(call.args[1] == 100 for call in client.pods.call_args_list)) + # Service-labelled backends are not exempt based on owner references. client.deployment.side_effect = deployments - client.pods.side_effect = [[pod(), stray]] - reason = gate.check_fleet(client, targets, (2, 15, 0), 100) - self.assertIn("stray", reason) + client.pods.side_effect = [[pod(), pod("v2.14.0", "stray")]] + self.assertIsNotNone(gate.check_fleet(client, targets, (2, 15, 0), 100)) + + def test_each_role_uses_its_own_expected_counts_and_zero_role_must_drain(self): + client = Mock() + targets = [target(replicas=2, min_ready=60), + target(replicas=0, name="automationwriter", + container="brainstore-automationwriter")] + reader = deployment(replicas=2, min_ready=60) + writer = deployment(replicas=0, container="brainstore-automationwriter") + writer["metadata"]["name"] = "automationwriter" + client.deployment.side_effect = [reader, writer] + client.pods.side_effect = [[pod(), pod(uid="second")], []] + self.assertIsNone(gate.check_fleet(client, targets, (2, 15, 0), 100)) + client.deployment.side_effect = [reader, writer] + client.pods.side_effect = [[pod(), pod(uid="second")], + [pod(container="brainstore-automationwriter")]] + self.assertIsNotNone(gate.check_fleet(client, targets, (2, 15, 0), 100)) + + def test_writer_waits_across_same_image_and_later_rollouts(self): + container = "brainstore-writer" + templates = (OLD_TEMPLATE, TEMPLATE, TEMPLATE, + LATER_TEMPLATE, LATER_TEMPLATE, LATER_TEMPLATE) + resources = [deployment(template=template, container=container) for template in templates] + for resource in resources: + resource["metadata"]["name"] = "writer" + resources[1]["status"]["updatedReplicas"] = 0 + resources[3]["status"]["updatedReplicas"] = 0 + old = pod(template=OLD_TEMPLATE, container=container) + current = pod(container=container) + later = pod(template=LATER_TEMPLATE, container=container) + initial = ordering(target(name="writer", container=container)) + latest = ordering(target(template=LATER_TEMPLATE, name="writer", container=container)) + client = api_client([initial, initial, initial, latest, latest, latest]) + client.deployment.side_effect = resources + client.pods.side_effect = [ + [old], [old, pod(is_ready=False, container=container)], + [current], [current, pod(template=LATER_TEMPLATE, is_ready=False, container=container)], + [later], [later], + ] + clock = FakeClock() + gate.wait_for_fleet(client, "release-ordering", (2, 15, 0), 120, 15, + clock.clock, clock.sleep, Mock()) + self.assertEqual(clock.now, 75) + + def test_new_config_does_not_accept_old_healthy_deployment_before_helm_update(self): + for expected in (target(template=LATER_TEMPLATE), target(replicas=2), + target(replicas=0), target(min_ready=60)): + with self.subTest(expected=expected): + client = api_client([ordering(expected), ordering(expected)]) + client.deployment.return_value = deployment() + client.pods.return_value = [pod()] + clock = FakeClock() + with self.assertRaises(TimeoutError): + gate.wait_for_fleet(client, "release-ordering", (2, 15, 0), 30, 15, + clock.clock, clock.sleep, Mock()) + + def test_completed_observation_cannot_release_application_after_deadline(self): + clock = FakeClock() + + def observe(name, deadline): + clock.sleep(30) + return deployment() + + client = api_client() + client.deployment.side_effect = observe + client.pods.return_value = [pod()] + with self.assertRaises(TimeoutError): + gate.wait_for_fleet(client, "release-ordering", (2, 15, 0), 60, 15, + clock.clock, clock.sleep, Mock()) + + +class MainTests(unittest.TestCase): + def environment(self): + return { + "BRAINSTORE_MINIMUM_VERSION": "2.15.0", + "ORDERING_CONFIGMAP": "release-ordering", + "BRAINSTORE_GATE_TIMEOUT_SECONDS": "60", + "BRAINSTORE_GATE_POLL_SECONDS": "15", + "POD_NAMESPACE": "custom", + } + + def test_uses_stable_configmap_and_fixed_minimum_version(self): + with patch.dict(os.environ, self.environment(), clear=True), \ + patch.object(gate, "KubernetesClient") as client, \ + patch.object(gate, "wait_for_fleet") as wait: + gate.main() + client.assert_called_once_with("custom", "/var/run/brainstore-gate") + wait.assert_called_once_with(client.return_value, "release-ordering", (2, 15, 0), 60, 15) + + def test_missing_or_invalid_environment_fails_before_polling(self): + invalid = { + "ORDERING_CONFIGMAP": (None, ""), + "BRAINSTORE_MINIMUM_VERSION": (None, "", "latest", "2.15.0-rc.1"), + "BRAINSTORE_GATE_TIMEOUT_SECONDS": (None, "", "invalid", "0", "-1", "15"), + "BRAINSTORE_GATE_POLL_SECONDS": (None, "", "invalid", "0", "-1", "60"), + } + for field, values in invalid.items(): + for value in values: + with self.subTest(field=field, value=value): + environment = self.environment() + if value is None: + del environment[field] + else: + environment[field] = value + with patch.dict(os.environ, environment, clear=True), \ + patch.object(gate, "wait_for_fleet") as wait, \ + self.assertRaises((KeyError, ValueError)): + gate.main() + wait.assert_not_called() class ClientTests(unittest.TestCase): @@ -275,6 +866,14 @@ def test_reloads_rotating_token_and_bounds_request_timeout(self): self.assertEqual(calls[1].args[0].get_header("Authorization"), "Bearer synthetic-token-2") self.assertEqual(calls[0].kwargs["timeout"], 2) + def test_expired_budget_does_not_start_an_api_request(self): + client = self.client("/synthetic") + client.opener = Mock() + with patch.object(gate.time, "monotonic", return_value=10), \ + self.assertRaises(TimeoutError): + client.get("/test", 10) + client.opener.open.assert_not_called() + def test_pod_lists_are_paginated(self): client = self.client("/synthetic") client.get = Mock(side_effect=[ diff --git a/tests/test_job_gate.py b/tests/test_job_gate.py new file mode 100644 index 0000000..ba11b3d --- /dev/null +++ b/tests/test_job_gate.py @@ -0,0 +1,256 @@ +import importlib.util +import io +import itertools +import json +import os +from pathlib import Path +import sys +import unittest +from unittest.mock import Mock, patch +import urllib.error + +FILES = Path(__file__).resolve().parents[1] / "braintrust/files" +# The chart mounts the shared runtime under these import names. +for name, file in [ + ("brainstore_gate", "brainstore-startup-gate.py"), + ("job_gate", "job-gate.py"), +]: + spec = importlib.util.spec_from_file_location(name, FILES / file) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) +gate = sys.modules["job_gate"] +shared = sys.modules["brainstore_gate"] + + +def job(*conditions, **metadata): + return {"metadata": metadata, "status": {"conditions": [ + {"type": condition, "status": "True"} for condition in conditions + ]}} + + +def ordering(name="release-migrate-" + "a" * 32): + return {"data": {"ordering.json": json.dumps({ + "job": name, + "targets": [{"deployment": "reader", "container": "brainstore-reader", + "template": "b" * 64, "replicas": 1, "minReadySeconds": 0}], + })}} + + +class Clock: + def __init__(self): + self.now = 0 + + def __call__(self): + return self.now + + def sleep(self, seconds): + self.now += seconds + + +class JobGateTests(unittest.TestCase): + def client(self, observations, configs=None): + observations = iter(observations) + configs = iter(configs) if configs is not None else itertools.repeat(ordering()) + + def get(path, deadline): + if "/configmaps/" in path: + resource = next(configs) + elif "/jobs/" in path: + resource = next(observations) + else: + raise AssertionError(f"unexpected API path: {path}") + if isinstance(resource, BaseException): + raise resource + return resource + + client = Mock(namespace="test-namespace") + client.get.side_effect = get + return client + + def wait(self, observations, timeout=20, configs=None): + clock = Clock() + client = self.client(observations, configs) + gate.wait_for_job(client, "release-ordering", timeout, 5, + clock=clock, sleep=clock.sleep, log=Mock()) + return client, clock.now + + def test_only_complete_condition_counts_as_success(self): + for resource in ({}, job(), job("SuccessCriteriaMet"), + {"status": {"succeeded": 1}}, + {"status": {"conditions": [{"type": "Complete", "status": "False"}]}}, + job("Complete", deletionTimestamp="now")): + with self.subTest(resource=resource): + self.assertFalse(shared.job_complete(resource)) + self.assertTrue(shared.job_complete(job("Complete"))) + + def test_failure_takes_precedence_over_completion_and_deletion(self): + for condition in ("Failed", "FailureTarget"): + for metadata in ({}, {"deletionTimestamp": "now"}): + with self.subTest(condition=condition, metadata=metadata): + with self.assertRaises(RuntimeError): + shared.job_complete(job(condition, "Complete", **metadata)) + + def test_malformed_job_observations_fail_closed(self): + for resource in (None, [], {"metadata": None}, {"status": None}, + {"status": {"conditions": None}}, + {"status": {"conditions": {}}}, + {"status": {"conditions": [None]}}, + {"status": {"conditions": ["Complete"]}}): + with self.subTest(resource=resource), self.assertRaises(ValueError): + shared.job_complete(resource) + + def test_exact_job_wait_retries_missing_job_rbac_and_server_errors(self): + errors = [urllib.error.HTTPError("url", code, "error", {}, io.BytesIO()) + for code in (404, 403, 500)] + client, elapsed = self.wait(errors + [job("Complete")]) + self.assertEqual(elapsed, 15) + self.assertEqual([call.args for call in client.get.call_args_list], [ + ("/api/v1/namespaces/test-namespace/configmaps/release-ordering", 20), + ("/apis/batch/v1/namespaces/test-namespace/jobs/release-migrate-" + "a" * 32, 20), + ] * 4) + + def test_retained_success_does_not_wait_for_another_observation(self): + client, elapsed = self.wait([job("Complete")]) + self.assertEqual(client.get.call_count, 2) + self.assertEqual(elapsed, 0) + + def test_running_job_then_success(self): + _, elapsed = self.wait([job(), job("Complete")]) + self.assertEqual(elapsed, 5) + + def test_current_config_job_is_selected_on_every_observation(self): + client, elapsed = self.wait([job(), job("Complete")], configs=[ + ordering("old-content-job"), ordering("new-content-job"), + ]) + self.assertEqual(elapsed, 5) + paths = [call.args[0] for call in client.get.call_args_list] + self.assertEqual(paths, [ + "/api/v1/namespaces/test-namespace/configmaps/release-ordering", + "/apis/batch/v1/namespaces/test-namespace/jobs/old-content-job", + "/api/v1/namespaces/test-namespace/configmaps/release-ordering", + "/apis/batch/v1/namespaces/test-namespace/jobs/new-content-job", + ]) + + def test_missing_config_does_not_reuse_previously_selected_job(self): + client, elapsed = self.wait([job(), job("Complete")], configs=[ + ordering("old-content-job"), {"data": {}}, ordering("new-content-job"), + ]) + self.assertEqual(elapsed, 10) + paths = [call.args[0] for call in client.get.call_args_list if "/jobs/" in call.args[0]] + self.assertEqual(paths, [ + "/apis/batch/v1/namespaces/test-namespace/jobs/old-content-job", + "/apis/batch/v1/namespaces/test-namespace/jobs/new-content-job", + ]) + + def test_missing_or_forbidden_ordering_config_retries_without_reading_job(self): + errors = [urllib.error.HTTPError("url", code, "error", {}, io.BytesIO()) + for code in (404, 403)] + client, elapsed = self.wait([job("Complete")], configs=errors + [ordering()]) + self.assertEqual(elapsed, 10) + self.assertEqual(client.get.call_count, 4) + self.assertEqual(sum("/jobs/" in call.args[0] for call in client.get.call_args_list), 1) + + def test_missing_or_malformed_config_never_allows_startup(self): + invalid_target = ordering() + config = json.loads(invalid_target["data"]["ordering.json"]) + config["targets"][0]["replicas"] = True + invalid_target["data"]["ordering.json"] = json.dumps(config) + for resource in ({}, {"data": {}}, {"data": {"ordering.json": "not JSON"}}, + ordering(""), invalid_target, + {"data": {"ordering.json": '{"job":"job","targets":[]}'}}): + with self.subTest(resource=resource): + client = self.client([], itertools.repeat(resource)) + clock = Clock() + with self.assertRaises(TimeoutError): + gate.wait_for_job(client, "release-ordering", 20, 5, + clock=clock, sleep=clock.sleep, log=Mock()) + self.assertTrue(all("/configmaps/" in call.args[0] + for call in client.get.call_args_list)) + + def test_running_missing_or_forbidden_job_never_allows_startup(self): + for observation in (job(), urllib.error.HTTPError("url", 404, "", {}, io.BytesIO()), + urllib.error.HTTPError("url", 403, "", {}, io.BytesIO())): + with self.subTest(observation=observation), self.assertRaises(TimeoutError): + self.wait(itertools.repeat(observation)) + + def test_failed_job_never_allows_startup(self): + with self.assertRaises(RuntimeError): + self.wait([job("Failed")]) + + def test_transient_observation_errors_retry(self): + client, elapsed = self.wait([OSError("network"), ValueError("bad JSON"), + job("Complete")]) + self.assertEqual(client.get.call_count, 6) + self.assertEqual(elapsed, 10) + + def test_observation_errors_do_not_disclose_response_details(self): + for config_failure in (False, True): + with self.subTest(config_failure=config_failure): + clock = Clock() + errors = itertools.repeat(ValueError("secret response body")) + client = self.client([], errors) if config_failure else self.client(errors) + log = Mock() + with self.assertRaises(TimeoutError) as raised: + gate.wait_for_job(client, "release-ordering", 6, 5, clock=clock, + sleep=clock.sleep, log=log) + self.assertNotIn("secret response body", str(raised.exception)) + self.assertNotIn("secret response body", str(log.call_args_list)) + + def test_job_path_escapes_current_job_name(self): + client, _ = self.wait([job("Complete")], configs=[ordering("name/other")]) + self.assertEqual(client.get.call_args.args[0], + "/apis/batch/v1/namespaces/test-namespace/jobs/name%2Fother") + + def test_late_success_does_not_escape_deadline(self): + clock = Clock() + client = Mock(namespace="ns") + + def late(path, deadline): + if "/configmaps/" in path: + return ordering() + clock.now = 21 + return job("Complete") + + client.get.side_effect = late + with self.assertRaises(TimeoutError): + gate.wait_for_job(client, "release-ordering", 20, 5, clock=clock, + sleep=clock.sleep, log=Mock()) + + +class MainTests(unittest.TestCase): + def environment(self): + return {"ORDERING_CONFIGMAP": "release-ordering", "JOB_TIMEOUT_SECONDS": "20", + "JOB_POLL_SECONDS": "5", "POD_NAMESPACE": "custom"} + + def test_uses_stable_ordering_configmap_not_fixed_job_environment(self): + with patch.dict(os.environ, self.environment(), clear=True), \ + patch.object(gate, "KubernetesClient") as client, \ + patch.object(gate, "wait_for_job") as wait: + gate.main() + client.assert_called_once_with("custom", "/var/run/migration-gate") + wait.assert_called_once_with(client.return_value, "release-ordering", 20, 5) + + def test_missing_or_invalid_environment_fails_before_polling(self): + invalid = { + "ORDERING_CONFIGMAP": (None, ""), + "JOB_TIMEOUT_SECONDS": (None, "", "invalid", "0", "-1", "5"), + "JOB_POLL_SECONDS": (None, "", "invalid", "0", "-1", "20"), + } + for field, values in invalid.items(): + for value in values: + with self.subTest(field=field, value=value): + environment = self.environment() + if value is None: + del environment[field] + else: + environment[field] = value + with patch.dict(os.environ, environment, clear=True), \ + patch.object(gate, "wait_for_job") as wait, \ + self.assertRaises((KeyError, ValueError)): + gate.main() + wait.assert_not_called() + + +if __name__ == "__main__": + unittest.main()