feat(slurm): cleanup stale node-local state before launch

[mikasenghaas]

Summary

• Add a pre-workload srun step to multi_node_rl.sbatch.j2, multi_node_sft.sbatch.j2, and inference.sbatch.j2 that runs once per allocated node and wipes stale state left over from prior jobs.
• Specifically kills orphan python/torchrun/vllm/prime_rl processes and removes IPC state under /dev/shm/vllm-*, /tmp/vllm-*, /tmp/torch-*, /tmp/torchelastic_*.
• Prints one line per node so the sbatch log confirms procs=N gpu_mem=MMiB on startup.

Motivation

When a SLURM job wedges in the CG (completing) state for a long time after scancel — which happens to us regularly on large multi-node RL runs that die in mid-NCCL/collective — SLURM doesn't always reap every process. The next job scheduled onto the same nodes inherits:

• orphan Python/vLLM/torchrun processes still holding GPU memory and sockets, and
• stale /dev/shm/vllm-* segments and /tmp/torchelastic_* rendezvous files.

The concrete failure mode we hit in production was decode engines hanging on startup with:

Waiting for READY message from DP Coordinator...

for 1800s until the orchestrator aborted. A manual pdsh cleanup of the allocated nodes (identical to what this PR adds inside the sbatch template) fixed it immediately on the next launch, and has continued to.

The cleanup is a no-op on clean nodes — it just prints procs=1 gpu_mem=0MiB.

Notes

• Relies on #SBATCH --exclusive, which all three templates already set, so we're only killing our own user's stale processes and never touching other users' workloads.
• Runs before srun's workload launch, so it can't race with the current job's own processes.
• Intentionally broad pkill -9 -f "vllm" etc. — anything matching prime_rl/torchrun/vllm in the process name on an exclusively-allocated node is by definition a leftover.

🤖 Generated with Claude Code

---

Note

Medium Risk
Adds an aggressive per-node preflight cleanup (pkill -9 and rm -rf in /dev/shm//tmp) to SLURM templates; while intended for --exclusive nodes, incorrect matching or environment assumptions could still terminate unintended processes or remove needed temp state.

Overview
Adds a pre-workload srun cleanup step to the inference, multi_node_rl, and multi_node_sft SLURM sbatch templates to proactively remove stale node-local state left behind by prior jobs.

The new step force-kills leftover python/torchrun/vllm/prime_rl processes (including comm-name matches like vllm::...), deletes related IPC/temp files under /dev/shm and /tmp, and logs a per-node [node-cleanup] line with remaining process count and GPU memory usage.

^{Reviewed by Cursor Bugbot for commit 11a4e7a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[S1ro1]

can we add vllm router processes as well?

[mikasenghaas]

Good call — added explicit vllm-router entries to both the pkill list and the procs-count regex in all three templates in 79e72b1. Functionally a no-op since the broader vllm pattern already matched vllm-router as a substring, but better to be explicit.

[mikasenghaas]

Good point — vllm::router is the kernel comm (prctl'd process name), not the cmdline, so pkill -f misses it. Just pushed 11a4e7a adding pkill -9 "vllm" and pkill -9 "vllm::.*" (no -f) to match against comm, plus broadened the procs count to ps -eo comm,args so we see these in the per-node status line.

[S1ro1]

lgtm 🐐

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 11a4e7a. Configure here.}

[cursor]

Cleanup script kills its own shell via pkill -f

High Severity

The srun bash -c '<script>' approach makes the entire script text part of the bash process's /proc/<pid>/cmdline. When pkill -9 -f "python.*prime_rl" (or any subsequent pkill -f pattern) runs as a child of that bash process, it matches the parent bash shell's own cmdline — which contains every pattern as a literal substring — and sends SIGKILL to it. This kills the cleanup shell on the very first pkill -f command, making the entire cleanup block non-functional. The job continues because there's no set -e, but no stale processes are killed, no temp files are removed, and no status line is printed.

Additional Locations (2)

• src/prime_rl/templates/multi_node_rl.sbatch.j2#L157-L173
• src/prime_rl/templates/multi_node_sft.sbatch.j2#L64-L80

 

^{Reviewed by Cursor Bugbot for commit 11a4e7a. Configure here.}