HPA and ArgoCD - scaling is disabled since the replica count of the target is zero

[KasparWe]

Hi there,

I've created a Deployment without setting replicas and created a ScaledObject for scaling the deployment based on number of messages in AWS SQS.
The replica count is set to 0 because the application should act event driven. So whenever a new message appears in the queue it will scale up/down.

The Problem
The HPA shows scaling is disabled since the replica count of the target is zero because the default value for replicas in the deployment is 0. => The ScaledObject will be in not active state

When setting replicas in the deployment to eg. to 1 the HPA will work as expected and the ScaledObject will show active state BUT ArgoCD`s AutoSync will fight till infinity against the difference from the deployment spec and terminate/create replicas .

Why the HPA requires the deployment to have a replica count set?

As a workaround I disabled the AutoSync for the Deployments which are scaled by the HPA but ArgoCD will show them as OutOfSync.

[zroubalik]

HPA doesn't properly support scaling to 0, but that's something that KEDA does. So once the Deployment is scaled to 0, KEDA will activate it to 1 replica, you shouldn't do that. So your original ask (scale 0<->N replicas based on AWS SQS) should work properly, without your interaction.

But there should be only one controller that manages the number of replicas in the Deployment, so in this case it is KEDA. If you have additional tool that tries to modify the replica number, it will result in conflicts. So in this case it is probably the ArgoCD AutoSync.

[ishworg]

As a workaround I disabled the AutoSync for the Deployments which are scaled by the HPA but ArgoCD will show them as OutOfSync.

All things considered, fwiw, disabling autosync in Argo is not a recommended approach (although different teams/orgs have differing deployment workflows). A better approach is to update Argo's application (declaratively in the YAML) - documented at https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#application-level-configuration

[ishworg]

Under https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#application-level-configuration, apps are deployed using a fixed number of replicas. After that, keda.sh's scaler will take over the replica's scale-in/scale-out activity and argo will completely ignore the JSON pointer that points to the deployment's replica count.

[JorTurFer]

In general, you shouldn't specify the replica count in the workload manifests when you work with autoscaling, because every time when you apply that manifest, the replica count will go to the fixed value in your manifest and then the autoscaling will put the desired replicas, but you could go from X to fixed value and the to X again, impacting in your users if the reduction is big

[ishworg]

@JorTurFer Sure. What you are describing is great but that's the basic minimum and general version of it. Most of the deployments are rollingUpdate and when you specify a fixed y for spec.replicas in your workload (for Deployment for e.g.,), the scheduler will ensure you have y replicas. However, rollingUpdate ensures all the health checks and surges are met before rolling out a new pod. Once this happens, your replica count is at y. The prior described scenario is where the responsibility of k8s scheduler ends. Now, comes the argo's own enforcement of y replica count (autosync+health enabled). This is where we need to tell argo to ignore y replica count (as specified in spec.replicas) and the above linked comment is what that achieves.

Now, during operational steady-state, keda.sh will take over the replica count completely and schedule replicas according to its own HPA policy driven by its own autoscaling policy (queue depth e.g.) using ScaledObject's min and max replicas.

With regards to:

X to fixed value and the to X again, impacting in your users if the reduction is big

Couple of things regarding service availability:

• Don't create one own's HPA (let Keda manage it)
• Peruse spec.advanced.restoreToOriginalReplicaCount set to false.
• Use a suitable stabilization window behaviour in ScaledObject.
• Use lifecycle hook in the container to manage availability https://github.com/kedacore/keda-docs/blob/main/content/docs/2.9/concepts/scaling-deployments.md#leverage-the-container-lifecycle

[dgrezza]

Is there a conclusion to managing KEDA with ArgoCD? since I got same issue

[Giaco9NN]

Hi. Keda's version is "2.18.0," and as a scalable target, I'm using Argo Rollouts v1.8.3. When starting from scratch, I try to create the rollout without specifying "spec.replicas" I received the error I sent you.

[JorTurFer]

Could you share an example of your workload and your SO? I'll try on my local

[Giaco9NN]

I'm working on it. I was not able to reproduce on a highly simplified scenario, but I can produce in a scenario with many components involved. I'm trying to simplify the scenario to simplify the debugging

[Giaco9NN]

Hi @JorTurFer. I found the issue and can confirm that Keda is not involved. The problem was a misconfiguration at the cluster level regarding pod disruption budgets and replica sets. Sorry for the confusion!

[JorTurFer]

No worries! happy to help :)