Skip to content

Listener Kubernetes Client throttling at 5 QPS #4554

Description

@FireGrey

Checks

Controller Version

0.14.2

Deployment Method

Helm

Checks

  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Request >5 jobs/second on the same scaleset for at least a brief duration

Describe the bug

The listener is effectively capped at being able to process 5 JobStarted messages per second once the burst bucket (10) is exhausted.
The controller exposes configuration for the k8s client via k8sClientRateLimiterQPS but this isn't available for the listener.
5 QPS is the default in the go k8s client when none is provided.
The kubernetes client in scaler.go doesn't pass in any configuration for the QPS or Burst.

Describe the expected behavior

The gha-runner-scale-set chart should include a setting similar to k8sClientRateLimiterQPS and k8sClientRateLimiterBurst like the controller does.

As a test I added the following to cmd/ghalistener/scaler/scaler.go:

	conf, err := rest.InClusterConfig()
	if err != nil {
		return nil, err
	}

+	conf.QPS = 20
+	conf.Burst = 30

	clientset, err := kubernetes.NewForConfig(conf)
	if err != nil {
		return nil, err
	}

Then requested a large number of jobs at the same time in GitHub.

Running the following prometheus query:

sum by(pod) (increase(gha_started_jobs_total{pod=~"<listener name>"}[1m]))

Before the change it would average ~290 jobs started per minute
After the change it would average ~1150 job started per minute

Additional Context

This is a hard limit on the number of jobs that can be started at a time per scaleset, even increasing the default would be greatly appreciated.

Controller Logs

N/A

Runner Pod Logs

N/A

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinggha-runner-scale-setRelated to the gha-runner-scale-set mode

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions