Repository navigation
Autoscale the AI Gateway by default #119
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| {{- $autoscaling := .Values.aiGateway.autoscaling | default dict }} | ||
| {{- if and .Values.aiGateway.enabled (dig "enabled" true $autoscaling) }} | ||
| {{- /* Bounds default from replicas so a custom count is not reset to 2-6. */}} | ||
| {{- $minReplicas := int (dig "minReplicas" (dig "replicas" 2 .Values.aiGateway) $autoscaling) }} | ||
| {{- $maxReplicas := int (dig "maxReplicas" (max 6 $minReplicas) $autoscaling) }} | ||
| {{- $cpuTarget := int (dig "cpu" "targetAverageUtilization" 70 $autoscaling) }} | ||
| {{- $memoryTarget := int (dig "memory" "targetAverageUtilization" 75 $autoscaling) }} | ||
| {{- $requests := (.Values.aiGateway.resources | default dict).requests | default dict }} | ||
| {{- if not (and (ge $minReplicas 1) (ge $maxReplicas $minReplicas) (gt $cpuTarget 0) (gt $memoryTarget 0) $requests.cpu $requests.memory) }} | ||
| {{- fail "aiGateway.autoscaling needs 1 <= minReplicas <= maxReplicas, positive CPU and memory targets, and aiGateway.resources.requests for CPU and memory" }} | ||
| {{- end }} | ||
| --- | ||
| apiVersion: autoscaling/v2 | ||
| kind: HorizontalPodAutoscaler | ||
| metadata: | ||
| name: {{ .Values.aiGateway.name }} | ||
| namespace: {{ include "braintrust.namespace" . }} | ||
| {{- with (merge (deepCopy .Values.aiGateway.labels) .Values.global.labels) }} | ||
| labels: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| spec: | ||
| scaleTargetRef: | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| name: {{ .Values.aiGateway.name }} | ||
| minReplicas: {{ $minReplicas }} | ||
| maxReplicas: {{ $maxReplicas }} | ||
| metrics: | ||
| - type: ContainerResource | ||
| containerResource: | ||
| name: cpu | ||
| container: ai-gateway | ||
| target: | ||
| type: Utilization | ||
| averageUtilization: {{ $cpuTarget }} | ||
| - type: ContainerResource | ||
| containerResource: | ||
| name: memory | ||
| container: ai-gateway | ||
| target: | ||
| type: Utilization | ||
| averageUtilization: {{ $memoryTarget }} | ||
| behavior: | ||
| scaleUp: | ||
| stabilizationWindowSeconds: 60 | ||
| scaleDown: | ||
| stabilizationWindowSeconds: 300 | ||
| {{- end }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # Simulate values retained by `helm upgrade --reuse-values` from a chart release | ||
| # that predates AI Gateway autoscaling. An explicit null removes the new chart | ||
| # default during Helm value coalescing. | ||
| aiGateway: | ||
| enabled: true | ||
| autoscaling: null |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,319 @@ | ||
| suite: AI Gateway autoscaling | ||
| templates: | ||
| - ai-gateway-hpa.yaml | ||
| - ai-gateway-deployment.yaml | ||
| - ai-gateway-configmap.yaml | ||
| values: | ||
| - __fixtures__/base-values.yaml | ||
| release: | ||
| namespace: braintrust | ||
| tests: | ||
| - it: renders no HPA when the gateway is disabled | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - hasDocuments: | ||
| count: 0 | ||
|
|
||
| - it: renders an HPA with AWS-parity defaults | ||
| set: | ||
| aiGateway.enabled: true | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - isKind: | ||
| of: HorizontalPodAutoscaler | ||
| - equal: | ||
| path: spec.scaleTargetRef.name | ||
| value: braintrust-ai-gateway | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 2 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 6 | ||
| - equal: | ||
| path: spec.metrics[0].containerResource | ||
| value: | ||
| name: cpu | ||
| container: ai-gateway | ||
| target: | ||
| type: Utilization | ||
| averageUtilization: 70 | ||
| - equal: | ||
| path: spec.metrics[1].containerResource | ||
| value: | ||
| name: memory | ||
| container: ai-gateway | ||
| target: | ||
| type: Utilization | ||
| averageUtilization: 75 | ||
|
|
||
| - it: lets the HPA own replicas by default | ||
| set: | ||
| aiGateway.enabled: true | ||
| template: ai-gateway-deployment.yaml | ||
| asserts: | ||
| - notExists: | ||
| path: spec.replicas | ||
|
|
||
| - it: applies custom bounds and targets | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.autoscaling.minReplicas: 4 | ||
| aiGateway.autoscaling.maxReplicas: 40 | ||
| aiGateway.autoscaling.cpu.targetAverageUtilization: 60 | ||
| aiGateway.autoscaling.memory.targetAverageUtilization: 80 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 4 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 40 | ||
| - equal: | ||
| path: spec.metrics[0].containerResource.target.averageUtilization | ||
| value: 60 | ||
| - equal: | ||
| path: spec.metrics[1].containerResource.target.averageUtilization | ||
| value: 80 | ||
|
|
||
| - it: keeps a custom replicas count as the floor | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.replicas: 4 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 4 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 6 | ||
|
|
||
| - it: raises maxReplicas to a replicas count above 6 | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.replicas: 10 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 10 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 10 | ||
|
|
||
| - it: raises maxReplicas to a minReplicas above 6 | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.autoscaling.minReplicas: 8 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 8 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 8 | ||
|
|
||
| - it: prefers explicit bounds over replicas | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.replicas: 10 | ||
| aiGateway.autoscaling.minReplicas: 3 | ||
| aiGateway.autoscaling.maxReplicas: 5 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 3 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 5 | ||
|
|
||
| - it: uses 2 replicas when replicas is null | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.replicas: null | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 2 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 6 | ||
|
|
||
| - it: rejects zero replicas | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.replicas: 0 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - failedTemplate: | ||
| errorMessage: aiGateway.autoscaling needs 1 <= minReplicas <= maxReplicas, positive CPU and memory targets, and aiGateway.resources.requests for CPU and memory | ||
|
|
||
| - it: rejects empty replicas | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.replicas: "" | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - failedTemplate: | ||
| errorMessage: aiGateway.autoscaling needs 1 <= minReplicas <= maxReplicas, positive CPU and memory targets, and aiGateway.resources.requests for CPU and memory | ||
|
|
||
| - it: rejects zero minReplicas | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.autoscaling.minReplicas: 0 | ||
|
Comment on lines
+165
to
+168
There was a problem hiding this comment.
The validation tests cover only one input shape per setting, leaving null/empty replica bounds, null utilization targets, and partial nested CPU/memory maps on reused values untested. These exercise different Helm coalescing and AGENTS.md reference: AGENTS.md:L14-L17 Useful? React with 👍 / 👎. |
||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - failedTemplate: | ||
| errorMessage: aiGateway.autoscaling needs 1 <= minReplicas <= maxReplicas, positive CPU and memory targets, and aiGateway.resources.requests for CPU and memory | ||
|
|
||
| - it: rejects maxReplicas below minReplicas | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.autoscaling.maxReplicas: 1 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - failedTemplate: | ||
| errorMessage: aiGateway.autoscaling needs 1 <= minReplicas <= maxReplicas, positive CPU and memory targets, and aiGateway.resources.requests for CPU and memory | ||
|
|
||
| - it: rejects an empty CPU target | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.autoscaling.cpu.targetAverageUtilization: "" | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - failedTemplate: | ||
| errorMessage: aiGateway.autoscaling needs 1 <= minReplicas <= maxReplicas, positive CPU and memory targets, and aiGateway.resources.requests for CPU and memory | ||
|
|
||
| - it: rejects a zero memory target | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.autoscaling.memory.targetAverageUtilization: 0 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - failedTemplate: | ||
| errorMessage: aiGateway.autoscaling needs 1 <= minReplicas <= maxReplicas, positive CPU and memory targets, and aiGateway.resources.requests for CPU and memory | ||
|
|
||
| - it: rejects a missing memory request | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.resources.requests.memory: null | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - failedTemplate: | ||
| errorMessage: aiGateway.autoscaling needs 1 <= minReplicas <= maxReplicas, positive CPU and memory targets, and aiGateway.resources.requests for CPU and memory | ||
|
|
||
| - it: renders no HPA when autoscaling is disabled | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.autoscaling.enabled: false | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - hasDocuments: | ||
| count: 0 | ||
|
|
||
| - it: keeps fixed replicas when autoscaling is disabled | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.autoscaling.enabled: false | ||
| aiGateway.replicas: 3 | ||
| template: ai-gateway-deployment.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.replicas | ||
| value: 3 | ||
|
|
||
| - it: uses default bounds for reused values from an older chart | ||
| values: | ||
| - __fixtures__/base-values.yaml | ||
| - __fixtures__/pre-ai-gateway-autoscaling-values.yaml | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 2 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 6 | ||
| - equal: | ||
| path: spec.metrics[1].containerResource.target.averageUtilization | ||
| value: 75 | ||
|
|
||
| - it: fills defaults around a partial --set on reused values | ||
| values: | ||
| - __fixtures__/base-values.yaml | ||
| - __fixtures__/pre-ai-gateway-autoscaling-values.yaml | ||
| set: | ||
| aiGateway.autoscaling.maxReplicas: 20 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 2 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 20 | ||
|
|
||
| - it: keeps a reused custom replicas count as the floor | ||
| values: | ||
| - __fixtures__/base-values.yaml | ||
| - __fixtures__/pre-ai-gateway-autoscaling-values.yaml | ||
| set: | ||
| aiGateway.replicas: 4 | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.minReplicas | ||
| value: 4 | ||
| - equal: | ||
| path: spec.maxReplicas | ||
| value: 6 | ||
|
|
||
| - it: keeps fixed replicas when reused values disable autoscaling | ||
| values: | ||
| - __fixtures__/base-values.yaml | ||
| - __fixtures__/pre-ai-gateway-autoscaling-values.yaml | ||
| set: | ||
| aiGateway.autoscaling.enabled: false | ||
| template: ai-gateway-deployment.yaml | ||
| asserts: | ||
| - equal: | ||
| path: spec.replicas | ||
| value: 2 | ||
|
|
||
| - it: renders the HPA on aws | ||
| values: | ||
| - __fixtures__/base-values.yaml | ||
| - __fixtures__/aws-values.yaml | ||
| set: | ||
| aiGateway.enabled: true | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - isKind: | ||
| of: HorizontalPodAutoscaler | ||
|
|
||
| - it: renders the HPA on azure | ||
| values: | ||
| - __fixtures__/base-values.yaml | ||
| - __fixtures__/azure-values.yaml | ||
| set: | ||
| aiGateway.enabled: true | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - isKind: | ||
| of: HorizontalPodAutoscaler | ||
|
|
||
| - it: renders the HPA on google without the AutoscalingMetric API | ||
| values: | ||
| - __fixtures__/base-values.yaml | ||
| - __fixtures__/google-values.yaml | ||
| set: | ||
| aiGateway.enabled: true | ||
| template: ai-gateway-hpa.yaml | ||
| asserts: | ||
| - isKind: | ||
| of: HorizontalPodAutoscaler | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add cross-template label-isolation coverage for both new resources: the current test only checks that disjoint component and global labels appear on one HPA, while neither the HPA nor PDB is tested for mutation leakage or conflicting-label precedence against other rendered resources. The repository explicitly requires a corresponding isolation test for every new template that merges labels.
AGENTS.md reference: AGENTS.md:L32-L35
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added HPA label-isolation tests in 0443689 (labels-merge-autoscaling_test.yaml): global/gateway precedence on the HPA, and gateway labels absent from another rendered component. The PDB was removed from this PR in a675059.