HPAs on QueryNodes prevent green/blue QueryNode deployments from completing

[jonathan-kron]

Summary:
We have a test cluster with HPAs on the querynode-0 and querynode-1 deployments with non-zero minReplicas. When I make a change to the CPU resources of the milvus.yaml file for the QueryNode component, it does a green/blue deployment and creates new pods for querynode-1, but it does not deallocate the pods for querynode-0.

Details:
QueryNode HPA Yaml
Milvus Yaml
Milvus Operator Logs
Milvus Exported Logs

kubectl apply -f [milvus-hpa.yaml]
kubectl apply -f [milvus.yaml]

I then edit the milvus CRD for this cluster to change the cpu resources on the querynodes.
I see 2 new querynodes come up in the querynode-1 deployment group
I never see the querynodes in querynode-0 drop off.

We've been seeing this in our production clusters as well.

[haorenfsa]

Hi @jonathan-kron, for now querynode HPA is not well supported during rollout.
I solution I can think of is that you need to manually delete HPA for querynode-0 and scale it gradually to 0. then restore the HPA.

ref:
#298