KEDA Kinesis Scaler – Metric Normalization always to 1

[josue-sanchez-wabtec]

While testing the Kinesis scaler with IRSA in EKS, I encountered multiple issues related to:

Unexpected metric normalization (value: "1" even with high lag)
Sensitivity to CloudWatch metric availability and lag generation

I am not seeing any error, but several behaviors are unclear or inconsistent with expectations and documentation.

I will really appreciate any comments from the community to understand more which CloudWatch metrics is keda kinesis scaler using. I red that possible GetRecords.IteratorAgeMilliseconds so that is why I was causing lag with a consumer, but no success.

Context:
KEDA Helm Chart: 2.19.0
Kubernetes: EKS (AWS)
Cloud Provider: AWS
Scaler: aws-kinesis-stream
Authentication: IRSA (pod identity)
Namespace: test-kinesis

These are the manifest I have used :

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kinesis-scaler
  namespace: test-kinesis
spec:
  scaleTargetRef:
    name: kinesis-consumer

  minReplicaCount: 1
  maxReplicaCount: 5

  pollingInterval: 10
  cooldownPeriod: 30

  triggers:
    - type: aws-kinesis-stream
      metadata:
        streamName: keda-test-stream
        awsRegion: us-east-1
        shardCount: "1"
        targetValue: "1000"
      authenticationRef:
        name: kinesis-auth

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: kinesis-auth
  namespace: test-kinesis
spec:
  podIdentity:
    provider: aws

{
  "Effect": "Allow",
  "Action": [
    "kinesis:DescribeStream",
    "kinesis:GetShardIterator",
    "kinesis:GetRecords",
    "kinesis:ListShards",
    "cloudwatch:GetMetricData",
    "cloudwatch:GetMetricStatistics"
  ],
  "Resource": "*"
}

Test Setup

Producer

while true; do
  aws kinesis put-record \
    --stream-name keda-test-stream \
    --partition-key "test" \
    --data "payload-$(date)" \
    --region us-east-1
done

Consumer

while true; do
  OUTPUT=$(aws kinesis get-records \
    --shard-iterator "$ITERATOR" \
    --region us-east-1)

  ITERATOR=$(echo $OUTPUT | jq -r '.NextShardIterator')

  sleep 10
done

Observed CloudWatch Metrics for around 15min:

{
    "Label": "GetRecords.IteratorAgeMilliseconds",
    "Datapoints": [
        {
            "Timestamp": "2026-05-19T21:16:00+00:00",
            "Average": 3232500.0,
            "Unit": "Milliseconds"
        },
        {
            "Timestamp": "2026-05-19T21:17:00+00:00",
            "Average": 2856666.6666666665,
            "Unit": "Milliseconds"
        },
        {
            "Timestamp": "2026-05-19T21:18:00+00:00",
            "Average": 2514250.0,
            "Unit": "Milliseconds"
        },
        {
            "Timestamp": "2026-05-19T21:15:00+00:00",
            "Average": 3687000.0,
            "Unit": "Milliseconds"
        },
        {
            "Timestamp": "2026-05-19T21:19:00+00:00",
            "Average": 274250.0,
            "Unit": "Milliseconds"
        }
    ]
}

Operator logs

2026-05-19T21:28:23Z	DEBUG	fallback	Fallback is not enabled, hence skipping the health update to the scaledobject	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler"}
2026-05-19T21:28:23Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler", "scaler": "awsKinesisStreamScaler", "metricName": "s0-aws-kinesis-keda-test-stream"}
2026-05-19T21:28:28Z	DEBUG	scale_handler	Getting metrics from trigger	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler", "trigger": "awsKinesisStreamScaler", "metricName": "s0-aws-kinesis-keda-test-stream", "metrics": [{"metricName":"s0-aws-kinesis-keda-test-stream","metricLabels":null,"timestamp":"2026-05-19T21:28:28Z","value":"1"}], "scalerError": null}
2026-05-19T21:28:28Z	DEBUG	fallback	Fallback is not enabled, hence skipping the health update to the scaledobject	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler"}
2026-05-19T21:28:28Z	DEBUG	grpc_server	Providing metrics	{"scaledObjectName": "kinesis-scaler", "scaledObjectNamespace": "test-kinesis", "metrics": "&ExternalMetricValueList{ListMeta:{   <nil>},Items:[]ExternalMetricValue{ExternalMetricValue{MetricName:s0-aws-kinesis-keda-test-stream,MetricLabels:map[string]string{},Timestamp:2026-05-19 21:28:28.719247801 +0000 UTC m=+116708.895870133,WindowSeconds:nil,Value:{{1000 -3} {<nil>}  DecimalSI},},},}"}
2026-05-19T21:28:33Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler", "scaler": "awsKinesisStreamScaler", "metricName": "s0-aws-kinesis-keda-test-stream", "metrics": [{"metricName":"s0-aws-kinesis-keda-test-stream","metricLabels":null,"timestamp":"2026-05-19T21:28:33Z","value":"1"}], "activity": true, "scalerError": null}
2026-05-19T21:28:33Z	DEBUG	fallback	Fallback is not enabled, hence skipping the health update to the scaledobject	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler"}
2026-05-19T21:28:33Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler", "scaler": "awsKinesisStreamScaler", "metricName": "s0-aws-kinesis-keda-test-stream"}
2026-05-19T21:28:43Z	DEBUG	scale_handler	Getting metrics from trigger	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler", "trigger": "awsKinesisStreamScaler", "metricName": "s0-aws-kinesis-keda-test-stream", "metrics": [{"metricName":"s0-aws-kinesis-keda-test-stream","metricLabels":null,"timestamp":"2026-05-19T21:28:43Z","value":"1"}], "scalerError": null}
2026-05-19T21:28:43Z	DEBUG	fallback	Fallback is not enabled, hence skipping the health update to the scaledobject	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler"}
2026-05-19T21:28:43Z	DEBUG	grpc_server	Providing metrics	{"scaledObjectName": "kinesis-scaler", "scaledObjectNamespace": "test-kinesis", "metrics": "&ExternalMetricValueList{ListMeta:{   <nil>},Items:[]ExternalMetricValue{ExternalMetricValue{MetricName:s0-aws-kinesis-keda-test-stream,MetricLabels:map[string]string{},Timestamp:2026-05-19 21:28:43.789248441 +0000 UTC m=+116723.965870772,WindowSeconds:nil,Value:{{1000 -3} {<nil>}  DecimalSI},},},}"}
2026-05-19T21:28:43Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler", "scaler": "awsKinesisStreamScaler", "metricName": "s0-aws-kinesis-keda-test-stream", "metrics": [{"metricName":"s0-aws-kinesis-keda-test-stream","metricLabels":null,"timestamp":"2026-05-19T21:28:43Z","value":"1"}], "activity": true, "scalerError": null}
2026-05-19T21:28:43Z	DEBUG	fallback	Fallback is not enabled, hence skipping the health update to the scaledobject	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler"}
2026-05-19T21:28:43Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "test-kinesis", "scaledObject.Name": "kinesis-scaler", "scaler": "awsKinesisStreamScaler", "metricName": "s0-aws-kinesis-keda-test-stream"}

Operator Metrics Server logs (Do not pay attention to the time, I have restarted these pods to just show useful logs and not error logs associated to a mis configuration at the ScaledObject I found earlier)

2026-05-19T21:31:42.947Z	INFO	keda_metrics_adapter	DEPRECATION WARNING: The '-v' flag is deprecated and will be removed in a future release. Please use '--zap-log-level' instead (e.g., '--zap-log-level=-1' for info, '--zap-log-level=-2' for debug)
2026-05-19T21:31:42.947Z	INFO	keda_metrics_adapter	DEPRECATION WARNING: The '--logtostderr' flag is deprecated and will be removed in a future release. Please use '--zap-encoder=console' instead
2026-05-19T21:31:42.947Z	INFO	keda_metrics_adapter	DEPRECATION WARNING: The '--stderrthreshold' flag is deprecated and has no equivalent in the new logging system. This flag is a no-op and will be removed in a future release
2026-05-19T21:31:42.965Z	INFO	keda_metrics_adapter	Starting metrics server
2026-05-19T21:31:42.969Z	INFO	keda_metrics_adapter	KEDA Version: 2.19.0
2026-05-19T21:31:42.970Z	INFO	keda_metrics_adapter	Git Commit: 3f88d60ae92f1174f67738f079f96ec17eeee77f
2026-05-19T21:31:42.970Z	INFO	keda_metrics_adapter	Go Version: go1.25.6
2026-05-19T21:31:42.970Z	INFO	keda_metrics_adapter	Go OS/Arch: linux/amd64
2026-05-19T21:31:42.970Z	INFO	keda_metrics_adapter	Running on Kubernetes 1.34	{"version": "v1.34.7-eks-40737a8"}
2026-05-19T21:31:42.971Z	INFO	keda_metrics_adapter	maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2026-05-19T21:31:42.972Z	INFO	keda_metrics_adapter	Connecting Metrics Service gRPC client to the server	{"address": "keda-operator.keda.svc.cluster.local:9666"}
2026-05-19T21:31:43.005Z	INFO	keda_metrics_adapter.provider	starting
2026-05-19T21:31:43.005Z	INFO	keda_metrics_adapter	starting adapter...
2026-05-19T21:31:43.026Z	INFO	keda_metrics_adapter.provider	Waiting for establishing a gRPC connection to KEDA Metrics Server
2026-05-19T21:31:43.026Z	INFO	keda_metrics_adapter	starting /metrics server endpoint
I0519 21:31:43.242415       1 secure_serving.go:211] Serving securely on 0.0.0.0:6443
I0519 21:31:43.242525       1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController
I0519 21:31:43.242567       1 shared_informer.go:349] "Waiting for caches to sync" controller="RequestHeaderAuthRequestController"
I0519 21:31:43.242639       1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/certs/tls.crt::/certs/tls.key"
I0519 21:31:43.242849       1 tlsconfig.go:243] "Starting DynamicServingCertificateController"
I0519 21:31:43.242976       1 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/certs/ca.crt"
I0519 21:31:43.243135       1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0519 21:31:43.243158       1 shared_informer.go:349] "Waiting for caches to sync" controller="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0519 21:31:43.343328       1 shared_informer.go:356] "Caches are synced" controller="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0519 21:31:43.343386       1 shared_informer.go:356] "Caches are synced" controller="RequestHeaderAuthRequestController"
2026-05-19T21:31:43.527Z	INFO	keda_metrics_adapter.provider	Connection to KEDA Metrics Service gRPC server has been successfully established	{"server": "keda-operator.keda.svc.cluster.local:9666"}

HPA events:

Name:                                                                                 keda-hpa-kinesis-scaler
Namespace:                                                                            test-kinesis
Labels:                                                                               app.kubernetes.io/managed-by=keda-operator
                                                                                      app.kubernetes.io/name=keda-hpa-kinesis-scaler
                                                                                      app.kubernetes.io/part-of=kinesis-scaler
                                                                                      app.kubernetes.io/version=2.19.0
                                                                                      scaledobject.keda.sh/name=kinesis-scaler
Annotations:                                                                          <none>
CreationTimestamp:                                                                    Fri, 15 May 2026 14:43:03 -0500
Reference:                                                                            Deployment/keda-kinesis-test
Metrics:                                                                              ( current / target )
  "s0-aws-kinesis-keda-test-stream" (target average value):  1 / 1
Min replicas:                                                                         1
Max replicas:                                                                         5
Deployment pods:                                                                      1 current / 1 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s0-aws-kinesis-keda-test-stream(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kinesis-scaler,},MatchExpressions:[]LabelSelectorRequirement{},})
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           <none>

[josue-sanchez-wabtec]

Interesting, seems the metric keda scaler is using is the shards counts, not directly related to CloudWatch.

HPA event traced that increase and therefore increased the pods based on the auto defined variable:

Name:                                                                                 keda-hpa-kinesis-scaler
Namespace:                                                                            test-kinesis
Labels:                                                                               app.kubernetes.io/managed-by=keda-operator
                                                                                      app.kubernetes.io/name=keda-hpa-kinesis-scaler
                                                                                      app.kubernetes.io/part-of=kinesis-scaler
                                                                                      app.kubernetes.io/version=2.19.0
                                                                                      scaledobject.keda.sh/name=kinesis-scaler
Annotations:                                                                          <none>
CreationTimestamp:                                                                    Fri, 15 May 2026 14:43:03 -0500
Reference:                                                                            Deployment/keda-kinesis-test
Metrics:                                                                              ( current / target )
  "s0-aws-kinesis-keda-test-stream" (target average value):  1 / 1
Min replicas:                                                                         1
Max replicas:                                                                         5
Deployment pods:                                                                      2 current / 2 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s0-aws-kinesis-keda-test-stream(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kinesis-scaler,},MatchExpressions:[]LabelSelectorRequirement{},})
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age    From                       Message
  ----    ------             ----   ----                       -------
  Normal  SuccessfulRescale  5m57s  horizontal-pod-autoscaler  New size: 2; reason: external metric s0-aws-kinesis-keda-test-stream(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kinesis-scaler,},MatchExpressions:[]LabelSelectorRequirement{},}) above target

Pods Increased to 2

[josue-sanchez-wabtec]

Second change done for the Stream shards. Created 2 more. From 2 to 4. Just to validate the trigger:

Name:                                                                                 keda-hpa-kinesis-scaler
Namespace:                                                                            test-kinesis
Labels:                                                                               app.kubernetes.io/managed-by=keda-operator
                                                                                      app.kubernetes.io/name=keda-hpa-kinesis-scaler
                                                                                      app.kubernetes.io/part-of=kinesis-scaler
                                                                                      app.kubernetes.io/version=2.19.0
                                                                                      scaledobject.keda.sh/name=kinesis-scaler
Annotations:                                                                          <none>
CreationTimestamp:                                                                    Fri, 15 May 2026 14:43:03 -0500
Reference:                                                                            Deployment/keda-kinesis-test
Metrics:                                                                              ( current / target )
  "s0-aws-kinesis-keda-test-stream" (target average value):  1 / 1
Min replicas:                                                                         1
Max replicas:                                                                         5
Deployment pods:                                                                      4 current / 4 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s0-aws-kinesis-keda-test-stream(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kinesis-scaler,},MatchExpressions:[]LabelSelectorRequirement{},})
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  32m   horizontal-pod-autoscaler  New size: 2; reason: external metric s0-aws-kinesis-keda-test-stream(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kinesis-scaler,},MatchExpressions:[]LabelSelectorRequirement{},}) above target
  Normal  SuccessfulRescale  42s   horizontal-pod-autoscaler  New size: 4; reason: external metric s0-aws-kinesis-keda-test-stream(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kinesis-scaler,},MatchExpressions:[]LabelSelectorRequirement{},}) above target

New pods created

Removing the shards and only keep two also scale things down to 2:

Name:                                                                                 keda-hpa-kinesis-scaler
Namespace:                                                                            test-kinesis
Labels:                                                                               app.kubernetes.io/managed-by=keda-operator
                                                                                      app.kubernetes.io/name=keda-hpa-kinesis-scaler
                                                                                      app.kubernetes.io/part-of=kinesis-scaler
                                                                                      app.kubernetes.io/version=2.19.0
                                                                                      scaledobject.keda.sh/name=kinesis-scaler
Annotations:                                                                          <none>
CreationTimestamp:                                                                    Fri, 15 May 2026 14:43:03 -0500
Reference:                                                                            Deployment/keda-kinesis-test
Metrics:                                                                              ( current / target )
  "s0-aws-kinesis-keda-test-stream" (target average value):  1 / 1
Min replicas:                                                                         1
Max replicas:                                                                         5
Deployment pods:                                                                      2 current / 2 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s0-aws-kinesis-keda-test-stream(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kinesis-scaler,},MatchExpressions:[]LabelSelectorRequirement{},})
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  48m   horizontal-pod-autoscaler  New size: 2; reason: external metric s0-aws-kinesis-keda-test-stream(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kinesis-scaler,},MatchExpressions:[]LabelSelectorRequirement{},}) above target
  Normal  SuccessfulRescale  16m   horizontal-pod-autoscaler  New size: 4; reason: external metric s0-aws-kinesis-keda-test-stream(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kinesis-scaler,},MatchExpressions:[]LabelSelectorRequirement{},}) above target
  Normal  SuccessfulRescale  21s   horizontal-pod-autoscaler  New size: 2; reason: All metrics below target

BY TE WAY !!!!

No CloudWatch permissions are needed if you are using IRSA approach, I removed them and the HPA was able to handle these two direction triggers (UP and DOWN) without issue.