ci: run bare metal maintenance on the right nodes - #2597
Merged
Conversation
sespiros
added
no changelog
sespiros
force-pushed
the
sse/bm-maintenance-runner-host
branch
3 times, most recently
from
August 21, 2026 08:46
7108310 to
9daf20c
Compare
sespiros
marked this pull request as ready for review
August 21, 2026 09:32
charludo
approved these changes
Aug 24, 2026
charludo
left a comment
charludo
left a comment
Collaborator
There was a problem hiding this comment.
Thanks! In general, LGTM. I'm not overly familiar with the available labeling strategies (both technically and best-practice) though, maybe Markus should also take a look at this?
sespiros
force-pushed
the
sse/bm-maintenance-runner-host
branch
from
August 24, 2026 16:52
9daf20c to
d495645
Compare
charludo
reviewed
Aug 25, 2026
charludo
left a comment
charludo
left a comment
Collaborator
There was a problem hiding this comment.
Thanks! LGTM, to more small things re: errors.
sespiros
force-pushed
the
sse/bm-maintenance-runner-host
branch
from
August 25, 2026 08:18
d495645 to
0a08926
Compare
burgerdev
approved these changes
Aug 27, 2026
burgerdev
left a comment
burgerdev
left a comment
Member
There was a problem hiding this comment.
Changing the jobs to target specific nodes makes perfect sense - there's just not a good way to run jobs but schedule like a daemonset that I know of.
However, I want to stress that having more than one main-runner was not how this was designed - the name itself is a giveaway.
- There should not be two action runners in one cluster, because it violates assumptions about concurrent execution. (containerd restarts will mess with concurrent tests)
- There should not be two nodes labeled
main-runner, because our CSI driver does not support that. - There's also the implicit assumption that the Github action runs on the node that's labeled
main-runner.
sespiros
force-pushed
the
sse/bm-maintenance-runner-host
branch
from
August 27, 2026 08:48
0a08926 to
824a2b1
Compare
Collaborator
Author
|
@burgerdev thanks, I implemented the fixes.
The second node didn't get the label by accident, iiuc this adds a contrast/internal/kuberesource/mutators.go Line 783 in 904f6e5 TestGPU deploy one workload per GPU model found in the cluster, so the B300 test can only be scheduled if that node carries the label.
|
The job garbage collects the nix store of the machine hosting the GitHub runner, but selected main-runner=true, which two nodes have carried since a second bare-metal machine joined the TDX-GPU cluster on 2026-08-14. That node needs main-runner so B300 test pods can be scheduled on it (#2566), so match the runner's own /etc/machine-id against the node's machineID instead, and require exactly one match, since cloned Scaleway pool images share a machine ID. It has timed out three times since, on 2026-08-18 and twice on 2026-08-20. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>
cleanup and cleanup-containerd select main-runner=true, which two nodes have carried since a second bare-metal machine joined the TDX-GPU cluster. They still pass, but each run cleans one of the two and leaves the other's runtime class directories, snapshotter state and images behind. Run the same job once per schedulable matching node, and fail when none is labelled rather than quietly cleaning nothing. The job is deleted before each apply because spec.template is immutable, so applying the next node's variant over one left by an interrupted run would fail. The job timeout grows with it: the worst case is one 600s wait per node. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>
kubectl wait only says "timed out waiting for the condition", and the next step deletes the namespace, so a failure takes its own evidence with it. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>
sespiros
force-pushed
the
sse/bm-maintenance-runner-host
branch
from
September 1, 2026 09:13
824a2b1 to
f061f3e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two nodes have carried
ci.contrast.edgeless.systems/main-runner=truesince a second bare-metal machine joined the TDX-GPU cluster on 2026-08-14, so the maintenance jobs pick one at random.nix-gccollects the runner's nix store and dies on the other machine (timed out 08-18 and twice on 08-20, blocking v1.23.1, hence b0753f7).It now resolves the runner's own node by matching
/etc/machine-idagainst the node'smachineID, so no labeling is needed.cleanupandcleanup-containerdpass but only clean the node they land on, so they now run once permain-runnernode, and a failure on one no longer skips the rest.Revert b0753f7 once the
bare metal maintenancerun this merge triggers is green on TDX-GPU.Why nix-gc dies on that node is still unknown, but the last commit dumps job and pod state on failure, so the next occurrence will say.