Fix: Skip replica management when HPA is attached

[niyomukiza-mechack]

What was changed

When an HPA exists via WorkerResourceTemplate the controller now skips enforcing the deployment's spec.Replicas of that version letting HPA manage the scaling.

Changes

I added a wew helper function hpaBuildIDSet() in planner.go It detects HPA-kind WRTs and marks all active build IDs as HPA managed
I added hpaBuildIDs parameter on getScaleDeployments. It skips spec.Replicas enforcement for versions in this set
Target version : gated on !hasHPA for spec.Replicas enforcement, but preserves the 0 - >1 bootstrap since HPAs cannot scale a Deployment from zero
I added Logs warning when spec.Replicas is set alongside an HPA, so a developer see the conflicting config instead of a silent fight

Why?

A developer configured a WorkerResourceTemplate with an HPA (minReplicas: 3) on a WorkerDeployment that also had spec.replicas: 1
The controller enforced replicas: 1 every 15 seconds, the HPA scaled back to 3, creating an endless loop
The root cause was getScaleDeployments enforced a WD's spec.Replicas on every reconcile without checking whether an HPA is already managing that Deployment's replica count.

Checklist

1. Closes [Bug] HorizontalPodAutoscaler continuously rescales due to controller interference #350

2. How was this tested:

Unit tests
Local e2e test (Kind cluster + Temporal Cloud)
Reproduced the bug and verified the fix on a Kind cluster
Before the fix, when I edited the WorkerDeployment's spec.Replicas to 1, the Controller locked replicas at 1 even thought I had an HPA with a minimum of 3 replicas and max 10. HPA was only used when WorkerDeployment's spec.Replicas was nil.

After this fix: Same spec.replicas: 1. Replicas stayed at 3 (HPA's value).

3. Any docs updates needed?

[jaypipes]

As of v1.8.1 of temporal-worker-controller, there is no ability to have a WorkerResourceTemplate per worker deployment version (i.e. "buildID"), so I don't see the need to return a map of buildIDs that are managed by an HPA. Just return true or false if an HPA is managing the scale for this WD.

[jaypipes]

See note above. This can just be a single boolean flag indicating whether HPA is managing the scaling or not for the WD as a whole.

[niyomukiza-mechack]

Hmh that makes sense, single boolean is much simpler

[carlydf]

A developer configured a WorkerResourceTemplate with an HPA (minReplicas: 3) on a WorkerDeployment that also had spec.replicas: 1
The controller enforced replicas: 1 every 15 seconds, the HPA scaled back to 3, creating an endless loop
The root cause was getScaleDeployments enforced a WD's spec.Replicas on every reconcile without checking whether an HPA is already managing that Deployment's replica count.

The behavior that you describe above is intentional and exactly what is documented on the workerDeploymentSpec.replicas field. We chose this because it is exactly how Kubernetes treats the spec.replicas field on the Deployment resource. See https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/#migrating-deployments-and-statefulsets-to-horizontal-autoscaling:

When an HPA is enabled, it is recommended that the value of spec.replicas of the Deployment and / or StatefulSet be removed from their manifest(s). If this isn't done, any time a change to that object is applied, for example via kubectl apply -f deployment.yaml, this will instruct Kubernetes to scale the current number of Pods to the value of the spec.replicas key. This may not be desired and could be troublesome when an HPA is active, resulting in thrashing or flapping behavior.

[carlydf]

Also, if we decide to merge this PR, it should not "close #350"

The problem reported in #350 says:

❌ Setting spec.replicas: null: Controller treats as 1

which is different from the behavior that this PR describes and changes.

(To be clear, I have not been able to reproduce the behavior described in #350, and the reporter hasn't gotten back to us with instructions to repro either, so I am not convinced we need to fix anything for #350)