Repository navigation
Kubernetes Cluster Scale Fails – Worker Nodes Not Joining After Scale Operation (CloudStack 4.19.0.1) #12565
Description
problem
We attempted to scale the cluster from 8 to 9 worker nodes using the Scale Cluster option in the CloudStack UI.
The new worker VM was created successfully and is in Running state in CloudStack. However, Kubernetes does not register the new node.
After the scale operation:
CloudStack shows 9 worker nodes
kubectl get nodes continues to show only 8 worker nodes
UI status stucks on scaling.
Error observed in CloudStack Management Server logs:
ERROR ... Unexpected exception while executing ScaleKubernetesClusterCmd
at KubernetesClusterResourceModifierActionWorker.removeSshFirewallRule
at KubernetesClusterScaleWorker.scaleKubernetesClusterIsolatedNetworkRules
versions
Environment details:
CloudStack version: 4.19.0.1
Cluster type: CloudStack Kubernetes Service (CKS)
Initial cluster size:
1 Control Plane
8 Worker Nodes (working fine)
Scale target: 9 Worker Nodes
Global setting cloud.kubernetes.cluster.max.size was increased from 10 to 50 prior to scaling.
The steps to reproduce the bug
Steps to Reproduce
Deploy a Kubernetes cluster using CloudStack Kubernetes Service (CKS) on CloudStack 4.19.0.1 with the following configuration:
1 Control Plane node
8 Worker nodes
From the CloudStack UI, navigate to:
Kubernetes → Clusters → → Scale Cluster
Scale the cluster by increasing the worker node count from 8 to 9 and submit the scale operation.
Observe the following behavior:
The new worker VM is created successfully and shows Running state in CloudStack.
The scale task stucks on scaling.
kubectl get nodes still showing node count as 8
What to do about it?
is there any workaround to resolve this without restarting the cluster
Activity
kiranchavala commentedon Feb 2, 2026
@vishnuvs369 Could you please let us know CKS iso details or which version Kubernetes version you have used also let us if the cks cluster was HA enabled
vishnuvs369 commentedon Feb 2, 2026
attaching screenshot of k8s iso image; also, HA is not enabled. having 1 control node and 8 worker nodes.
kiranchavala commentedon Feb 2, 2026
@vishnuvs369 Thanks, could you please provide the entire management server.log
vishnuvs369 commentedon Feb 2, 2026
Please find the attached entire management server log
management-server.zip
kiranchavala commentedon Feb 2, 2026
@vishnuvs369 Not hitting the issue on 4.22, was able to scale the cks cluster successfully
global setting set to cloud.kubernetes.cluster.max.size to 10
vishnuvs369 commentedon Feb 2, 2026
@kiranchavala Thank you for the update.
We do have plans to upgrade to the latest CloudStack version in the future. However, at the moment, upgrading or restarting the cluster is not feasible in our environment.
In the meantime, could you please advise if there is any workaround or recommended approach to scale the CKS cluster on CloudStack 4.19.0.1 without requiring a cluster restart?
Your guidance would be greatly appreciated.
Pearl1594 commentedon Feb 2, 2026
@vishnuvs369 For the give CKS cluster, does the cluster network's source NAT IP have any firewall rules in place? Based on the logs you've shared, it shows that there's an NPE when attempting to delete the firewall rules in place:
kiranchavala commentedon Feb 3, 2026
@vishnuvs369 please check the firewall and portforwarding rules of the cks networks( source nat IP)
Also you stop the cks cluster and change the service offering if that solves your use case
vishnuvs369 commentedon Feb 3, 2026
@kiranchavala @Pearl1594
Attaching the screenshots of firewall and portforwarding rules for the CKS cluster.
As it is a production cluster, we wont able to stop it now...is there any other workaround.