BuildKit autoscaling (staging + prod): in-cluster KEDA + LB queue + warm baseline

[huydhn]

Autoscale the per-arch buildkitd pools so ciflow/docker bursts don't overload the existing pods, scaling back to a small warm baseline when idle. Opt-in via buildkit.autoscaling.enabled; clusters without it are unchanged.

How

• One build per pod — HAProxy server maxconn 1 (matches max-parallelism=1) + timeout queue 60m. Excess builds queue and flow onto new pods as they register, instead of stacking on busy pods — so scaled-up pods don't sit idle (the burst problem).
• In-cluster scale signal — KEDA ScaledObject per arch via metrics-api on the LB's own metrics (haproxy_backend_current_sessions). No Grafana / external metrics backend.
• Warm baseline — amd64_min=2 / arm64_min=4 ≈ 1 physical node each; *_max caps the burst and sizes the NodePool limits.
• Kill-free scale-down — preStop drain (waits for the pod's :1234 to go idle) + PDB + long terminationGracePeriodSeconds.

Needs the keda module (CRDs).

Enabled on

• Staging (arc-staging): amd64 m6id.24xlarge @ 2/node, arm64 m7gd.16xlarge @ 4/node. min 2 / 4, max 8 / 8.
• Prod (arc-cbr-production): same instances/min. Max sized from 14-day docker-build concurrency — amd64 128 (peak 105 + headroom), arm64 16 (peak 8, likely capped by the old fixed pool). Replaces the old fixed 32 / 8.

Integration test

osdc/integration-tests gains a scale test (build-image-scale.yaml): bursts 8 parallel buildctl builds per arch, each holding a maxconn=1 slot ~10m via a sleep. With min=2 the builds serialize (~43m) and the job times out (30m) unless KEDA scales up (~18m, one wave) — so the suite fails if autoscaling doesn't happen.

Companion change

A runner-side connect retry in pytorch/pytorch .ci/docker/build.sh lets a build wait for a pod from a cold/queued pool (drafted separately).

just lint 13/13 · just test pass.

[jeanschmidt]

this does not solve the burst problem, when a change comes and trigger docker rebuild on multiple CIs, all the connections will be dispatched to the current existing buildkit pods and the newer ones will sit idle. We need to find a way to fix this.

[huydhn]

Staging validation run

Drove a balanced burst of 8 amd64 + 8 arm64 builds against the staging pool (each held a maxconn=1 slot ~5m), the same shape as the new integration-test scale test: https://github.com/pytorch/pytorch-canary/actions/runs/27247502866 — 16/16 builds succeeded.

BuildKit nodes / pods during the run:

Phase	amd64 pods	arm64 pods	buildkit nodes
Baseline (before)	2	4	2 (1 amd64 + 1 arm64)
Mid-burst (~T+9m)	scaling 2→8	scaling 4→8	climbing; 1 pod already draining
Peak	8	8	6 (4× m6id.24xlarge + 2× m7gd.16xlarge)
After (~T+18m)	back to 2	back to 4	trailing down

Observations:

• Queue worked as intended — every queued buildctl connected and rode the queue onto new pods as they registered; no connect timeouts, so no runner-side wait was needed for this burst.
• Scale-up — KEDA brought both arches to max (8/8) off the in-cluster HAProxy session metric.
• Kill-free scale-down — pods drained (preStop waited for :1234 to go idle) and returned to the 2/4 baseline with zero failed builds.
• Node consolidation lag (expected) — with consolidationPolicy: WhenEmpty, survivor pods left some nodes half-full, so the node count trails the pod count back down rather than dropping immediately. This is the deliberate trade documented in the module README (some idle node cost for zero build disruption).

[huydhn]

Closing this in favour of the stack at #725