5k scale test: 1-master vs 3-master HA comparison

[Jefftree]

HA improves performance, but surfaces an interesting behavior with the node authorizer graph.

Comparison of the 5000-node scale test on a 1-master vs a 3-master control plane (see details at bottom). Serves as additional data point for promoting HA test to release-blocking: #37360

Throughput

Barrier	1-master	3-master HA
Waiting for 'create objects'	38.2 min	15.3 min	-60%
Job wall clock	119.4 min	75.8 min	-37%

Differences

Load phase	1-master	3-master HA
Peak watches, busiest apiserver	174,476	106,612	1.6x lower
Peak goroutines, busiest	385,559	232,634	1.7x lower
Peak heap, busiest	55.4 GB	40.3 GB	-27%
Watch events delivered, busiest apiserver	729 M	340 M	2.1x less fan-out
Peak apiserver request rate	9,329 /s	15,879 /s	1.7x
apiserver->etcd p99	0.20-0.70 s	0.03-0.10 s	~5x lower
APF rejects (all apiservers, whole run)	1,098,233	240,336	4.6x fewer
APF system queue peak, busiest apiserver	3,707	3,118	-16%
Terminated watchers	38,039	17,276	2.2x fewer
403s (node authorizer)	100	78,848	790x worse
WATCHLIST configmaps + secrets (served syncs)	422,979	399,121	unchanged
LIST configmaps + secrets (fallbacks)	8,120	27,535	3.4x more
-> fallback rate	1.9%	6.5%	3.4x

Node Authorizer 403s and WATCHLIST fallback

Kubelets get 403 no relationship found when fetching their own pods' configmaps, secrets and SA tokens. The single-master run sees 100 of these across the whole test, the HA run sees 78,848.

Each apiserver builds its own node authorizer graph from its own pod informer, so the three graphs are independently stale and disagree for a short window after every pod binding, while a single master has one graph and cannot disagree with itself.

The flow:

1. The scheduler binds pod P to node N.
2. The kubelet on N learns about P from its node-filtered pod watch, served by apiserver A, and starts fetching what P mounts: a configmap, a secret, an SA token.
3. Kubelets are not pinned to one apiserver, so requesting those other resources can land on a different apiserver, B. 2,584 of 5,004 nodes talked to more than one apiserver during the run.
4. B authorizes it against B's own node authorizer graph, which B builds from B's own pod informer. If B has not seen pod P yet, there is no node -> pod -> secret edge, so B denies with 403.
5. The kubelet reflector's watch-list fails and it falls back to a plain LIST, retrying until B catches up. B catches up within seconds, so the denial is a window, not a permanent failure.

The 403 causes the WATCHLIST to fail and fall back to a LIST. The same number of kubelet configmap/secret syncs happen in both runs (423k vs 399k), but HA fails 6.5% of them on the first try versus 1.9%. The fallback LISTs are single-object reads (~940 bytes), so the actual cost is very low, but it appears as metric inflation.

Full time series

x = minutes since load-phase start. Red = 1-master, blue = 3-master HA. At minute 19, apiserver restarts and its watches get redistributed to the other apiservers.

Runs compared

	1-master	3-master HA
Build	2075264067504705536 (07-09)	2075807663861534720 (07-11)
Control plane	1 x c4-standard-96	3 x c4-standard-96
etcd	1 member	3 members
Extra variant	-	apiserver restart mid-test

/sig scalability

[Jefftree]

/cc @serathius @p0lyn0mial @michaelasp

[michaelasp]

/cc @liggitt as well

[michaelasp]

Curious why we see such a high spike compared to single master. At worst I would assume that we might get the worst of the 3 masters from kubelet, but this seems to be because of us not using the same master for state change/ action. The out of sync informers between the 3 masters cause this to crop up a lot more frequently.

The kubelet reflector's watch-list fails and it falls back to a plain LIST, retrying until B catches up. B catches up within seconds, so the denial is a window, not a permanent failure.

While not a problem here I could see this causing issues for some workloads, though maybe not as bad as other cases since we usually just list/watch a single secret/configmap/etc...

Let me see if there's a way to mitigate this from the apiserver side, though despite the alarming increase it still seems ~ok?

[Jefftree]

B catches up within seconds

well this assumption on my part was wrong. With the new resourceVersion metrics you added, it looks like one of the instances was quite behind and had to relist multiple times. Most of the rejections were from that. Hard to see what exactly went wrong with that instance, load was distributed pretty evenly, had plenty of headroom in terms of utilization (27/96 CPU core peak) , and other apiservers were healthy. HA mode's increased throughput does causing additional strain, and the apiserver was colocated with kube-scheduler leader, but nothing suggests that the apiserver was starved on resources in any way.

[michaelasp]

Huh, do we see the other metrics spike such as the fifo queue depth, that would indicate stalling on the client side. If it's not CPU bound is there maybe some slow lock?

[p0lyn0mial]

If I read the previous graph correctly, then the pod informer used by the node authorizer was behind. If that's true (after checking the root cause) I wonder whether it would be feasible to detect pod informer staleness and delay or "throttle" responses to kubelets.

[michaelasp]

I was discussing this with @liggitt previously. I think one of the issues is that authorizors don't have a backoff response, they only either allow, deny, or no decision. We'd need to make some api change to allow for another type of response there. That would be client based though, which expects the clients to act nicely. I wonder if there's something we can do that allows us to throttle on the apiserver end by detecting our own staleness, though that gets tricky.

[serathius]

Differences

Load phase	1-master	3-master HA
Peak watches, busiest apiserver	174,476	106,612	1.6x lower

Can you clarify if 100k is per VM or sum?

[serathius]

Peak apiserver request rate 9,329 /s 15,879 /s 1.7x

Can you split by request type?

[serathius]

apiserver->etcd p99 0.20-0.70 s 0.03-0.10 s ~5x lower

What type of request it is?

[serathius]

Terminated watchers 38,039 17,276 2.2x fewer

Number of terminated watches has huge variance, please use more samples than 1 when doing comparison.

[serathius]

1-master 3-master HA
Build 2075264067504705536 (07-09) 2075807663861534720 (07-11)
Control plane 1 x c4-standard-96 3 x c4-standard-96
etcd 1 member 3 members
Extra variant - apiserver restart mid-test

Please disable apiserver restart from comparison. Let's not compare apples to oranges.

[serathius]

I was discussing this with @liggitt previously. I think one of the issues is that authorizors don't have a backoff response, they only either allow, deny, or no decision. We'd need to make some api change to allow for another type of response there. That would be client based though, which expects the clients to act nicely. I wonder if there's something we can do that allows us to throttle on the apiserver end by detecting our own staleness, though that gets tricky.

@michaelasp Can we get some plan for fixing this? Seems that this is the biggest cause of scalability instability (excluding recent infra issues).