Checks
Controller Version
0.14.2
Deployment Method
Helm
Checks
To Reproduce
Request >5 jobs/second on the same scaleset for at least a brief duration
Describe the bug
The listener is effectively capped at being able to process 5 JobStarted messages per second once the burst bucket (10) is exhausted.
The controller exposes configuration for the k8s client via k8sClientRateLimiterQPS but this isn't available for the listener.
5 QPS is the default in the go k8s client when none is provided.
The kubernetes client in scaler.go doesn't pass in any configuration for the QPS or Burst.
Describe the expected behavior
The gha-runner-scale-set chart should include a setting similar to k8sClientRateLimiterQPS and k8sClientRateLimiterBurst like the controller does.
As a test I added the following to cmd/ghalistener/scaler/scaler.go:
conf, err := rest.InClusterConfig()
if err != nil {
return nil, err
}
+ conf.QPS = 20
+ conf.Burst = 30
clientset, err := kubernetes.NewForConfig(conf)
if err != nil {
return nil, err
}
Then requested a large number of jobs at the same time in GitHub.
Running the following prometheus query:
sum by(pod) (increase(gha_started_jobs_total{pod=~"<listener name>"}[1m]))
Before the change it would average ~290 jobs started per minute
After the change it would average ~1150 job started per minute
Additional Context
This is a hard limit on the number of jobs that can be started at a time per scaleset, even increasing the default would be greatly appreciated.
Controller Logs
Runner Pod Logs
Checks
Controller Version
0.14.2
Deployment Method
Helm
Checks
To Reproduce
Describe the bug
The listener is effectively capped at being able to process 5 JobStarted messages per second once the burst bucket (10) is exhausted.
The controller exposes configuration for the k8s client via
k8sClientRateLimiterQPSbut this isn't available for the listener.5 QPS is the default in the go k8s client when none is provided.
The kubernetes client in
scaler.godoesn't pass in any configuration for the QPS or Burst.Describe the expected behavior
The
gha-runner-scale-setchart should include a setting similar tok8sClientRateLimiterQPSandk8sClientRateLimiterBurstlike the controller does.As a test I added the following to
cmd/ghalistener/scaler/scaler.go:conf, err := rest.InClusterConfig() if err != nil { return nil, err } + conf.QPS = 20 + conf.Burst = 30 clientset, err := kubernetes.NewForConfig(conf) if err != nil { return nil, err }Then requested a large number of jobs at the same time in GitHub.
Running the following prometheus query:
Before the change it would average ~290 jobs started per minute
After the change it would average ~1150 job started per minute
Additional Context
This is a hard limit on the number of jobs that can be started at a time per scaleset, even increasing the default would be greatly appreciated.Controller Logs
Runner Pod Logs