Align kops scalability configuration with kube-up

[serathius]

/kind feature

After #18069 I decided to look deeper into what other configuration has potentially changed with kubernetes/test-infra#36137.

Came up with the following table, we should take a look and consider aligning configuration.

Component	Configuration Option	kube-up (test-infra)	kops (run-test.sh)	Kubernetes Default	Status / Notes
Kube-APIServer	Log Level	3	2	0	Decreased (3 -> 2).
Kube-APIServer	Max Inflight	640	800	400	Increased (640 -> 800).
Kube-APIServer	Delete Workers	16	100	1	Increased (16 -> 100).
Kube-APIServer	Compaction Interval	150s	N/A	300s	Increased (150s -> 300s)
Controller Manager	QPS	100	500	20	Increased (100 -> 500).
Controller Manager	Burst	100	500	30	Increased (100 -> 500).
Controller Manager	Concurrent Node Syncs	N/A	10	5	Increased ( 5 -> 10)
Kube-Scheduler	QPS	100	500	50	Increased (100 -> 500).
Kube-Scheduler	Burst	100	500	100	Increased (100 -> 500).
Kubelet	API QPS	100	N/A	50	Decreased ( 100 -> 50)
Kubelet	Max Pods	N/A	96	110	Decreased (110 -> 96)

/cc @upodroid @mborsz @wojtek-t @hakman

[ronaldngounou]

Thank you for putting this together.
One approach is to update the values in test-infra to the default kubernetes values. They don't have any influence on the testgrid.
Obviously, modifying the kops values would impact the testgrid results.

[serathius]

Kops has influence. https://github.com/kubernetes/kops/blob/master/tests/e2e/scenarios/scalability/run-test.sh

[ronaldngounou]

To clarify, only the values in the kops column are the ones that are used to run the tests. Example from test execution: https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-e2e-kops-aws-scale-amazonvpc-using-cl2/2033076091358285824/build-log.txt

The values in the test-infra column and kubernetes-default column are not used, therefore are not affecting scalability tests. One way to uniformize the values would be to match the kubernetes-default in the test-infra. What do you think?

Once we agree, I'm willing to pick this issue.

[serathius]

The values in the test-infra column and kubernetes-default column are not used, therefore are not affecting scalability tests.

Please read the top comment, they used to be used before migrating to kops, and they were changed without consideration. They were result of years of tuning scalability testing so we double check if they are important.

Once we agree, I'm willing to pick this issue.

That's great, before that let's wait for feedback about if the variables changes could be problematic.

[mborsz]

I'm sure that changing values like log level, qps, delete workers have performance and scalability implications.

We should revert the configuration to what we had in kube-up setup. If we believe that some other values should be used, this should be explicit decision based on some benchmarks and tests.

[serathius]

@ronaldngounou can you prepare PR to update https://github.com/kubernetes/kops/blob/master/tests/e2e/scenarios/scalability/run-test.sh

[ronaldngounou]

If we believe that some other values should be used, this should be explicit decision based on some benchmarks and tests.

• Benchmark and tests analysis for deleteCollectionsWorkers = 100 in kops: AWS KOPS Load Test - ec2-master-scale-performance failing due to not meeting "Delete event SLOs" kubernetes#135737 (comment)

• Some historical context on KOPS_SCHEDULER_QPS and KOPS_SCHEDULER_BURST: https://github.com/kubernetes/test-infra/pull/25533/changes#diff-1dc2bb0496cbc283793e5d8fd46759b6ced4f9be52041cb909e3bf1b097ec78fR117

it should allow to hugely decrease pod-startup-latency across the whole test

[ronaldngounou]

/assign ronaldngounou