Prerequisites
Code of Conduct
Feature Summary
Fault Quarantine and Labeler each write node objects with a GET followed by a full PUT. The GET exists solely to fetch resourceVersion for conflict detection. PATCH doesn't need it — it applies only the changed fields directly against etcd. Switching halves the API calls per node write and doubles throughput at any QPS setting.
Problem/Use Case
Both components follow the same pattern: GET the node to obtain resourceVersion, mutate the object in memory, then PUT the entire object back. This costs 2 API calls per node write.
PUT (client-go's Update()) requires a resourceVersion to detect write conflicts. Without a prior GET, the API server rejects the PUT. PATCH with application/merge-patch+json applies only the changed fields — no resourceVersion needed, no prior GET.
FQ cordons nodes by setting taints and Unschedulable. Labeler writes GPU labels. Both own distinct field namespaces — nothing else writes FQ's taints or Labeler's nvidia.com/* labels concurrently — so PATCH conflict detection is not needed.
At default QPS=5 with 2 calls per node:
- FQ cordons at 2.5 nodes/s
- Labeler labels at ~1 node/s → cold-relabel of a 100k node fleet takes ~28 hours
Proposed Solution
Replace Nodes().Get() + Nodes().Update() with Nodes().Patch() using types.MergePatchType. Send only the fields being changed — taints + spec.unschedulable for FQ; the label map for Labeler.
Relevant files:
fault-quarantine/pkg/informer/k8s_client.go:61-68labeler/pkg/labeler/labeler.go:626,682
Expected improvement: API calls per node write drop from 2 to 1. FQ: 2.5 → 5 nodes/s. Labeler: ~1 → ~2 nodes/s, cutting cold-relabel time from ~28h to ~14h at 100k nodes.
Component
Fault Management
Prerequisites
Code of Conduct
Feature Summary
Fault Quarantine and Labeler each write node objects with a GET followed by a full PUT. The GET exists solely to fetch
resourceVersionfor conflict detection. PATCH doesn't need it — it applies only the changed fields directly against etcd. Switching halves the API calls per node write and doubles throughput at any QPS setting.Problem/Use Case
Both components follow the same pattern: GET the node to obtain
resourceVersion, mutate the object in memory, then PUT the entire object back. This costs 2 API calls per node write.PUT(client-go'sUpdate()) requires aresourceVersionto detect write conflicts. Without a prior GET, the API server rejects the PUT.PATCHwithapplication/merge-patch+jsonapplies only the changed fields — noresourceVersionneeded, no prior GET.FQ cordons nodes by setting taints and
Unschedulable. Labeler writes GPU labels. Both own distinct field namespaces — nothing else writes FQ's taints or Labeler'snvidia.com/*labels concurrently — so PATCH conflict detection is not needed.At default QPS=5 with 2 calls per node:
Proposed Solution
Replace
Nodes().Get()+Nodes().Update()withNodes().Patch()usingtypes.MergePatchType. Send only the fields being changed — taints +spec.unschedulablefor FQ; the label map for Labeler.Relevant files:
fault-quarantine/pkg/informer/k8s_client.go:61-68labeler/pkg/labeler/labeler.go:626,682Expected improvement: API calls per node write drop from 2 to 1. FQ: 2.5 → 5 nodes/s. Labeler: ~1 → ~2 nodes/s, cutting cold-relabel time from ~28h to ~14h at 100k nodes.
Component
Fault Management