make sure we test LoadBalancers at scale

[danwinship]

#36720 removes the LoadBalancer tests from gce-master-scale-correctness because that job is actually too large for the tests to reliably work.

We should still be regularly testing LoadBalancers at 1000 or 2000-node scale though.

(Also, when we say "testing LoadBalancers", we really mean "testing cloud-provider-gcp", because as discussed in #36720, in kube-proxy load balancer services don't scale any differently than any other kind of services; it's only the cloud provider where we need the extra testing.)

/sig scalability
/sig network
/sig cloud-provider
/area provider/gcp

cc @wojtek-t @aojea @bowei

[Dipanshusinghh]

@danwinship Hi, I want to work on this issue.
I was going through #36720 to understand why the LoadBalancer tests were removed from gce-master-scale-correctness.

I’m still exploring how the large scale jobs are setup currently in test-infra.
Should this be handled by creating a seperate dedicated job for LoadBalancer testing, or by extending some existing scalability job for better cloud-provider-gcp coverage?

Any pointers or suggestions would really help. Thanks!

[danwinship]

Those are really more questions for sig-scalability and the provider-gcp subproject of sig-cloud-provider.

There may already be 1000-node or 2000-node GCP jobs being run, in which case we just need to make sure that they are running the LB tests.

[Dipanshusinghh]

Hi @danwinship.

I spent some time digging through the current test-infra configs to understand whats already running theree.

I checked the jobs mainly under config/jobs/kubernetes/sig-scalability/ to see if there is any existing 1000 or 2000-node GCP job we could reuse for LoadBalancer tests. From what I can see, there is no proper 1000-node correctness job right now. Most of the existing scale setups are either around 100 nodes or directly jump to 5000 nodes. And like you mentioned in #36720, the LoadBalancer tests were already removed from the 2000-node gce-master-scale-correctness job because of flakiness.

Since 2000-node seems too unstable for this and there is no 1000-node job currently available, I feel the cleanest option here could be:-

Proposal: create a new dedicated 1000-node periodic job (something like ci-kubernetes-e2e-gce-scale-lb-1000) which only focuses on cloud-provider-gcp LoadBalancer tests. This way it stays isolated and doesn't interfere with other scalability tests.

@wojtek-t @aojea @bowei — from sig-scalability / provider-gcp side, would this approach make sense? Or is there any existing job I might have missed where we can safely enable these tests without running into the same flakiness issues we saw at 2000 nodes?

If this sounds okay, I can start working on a PR for it