Currently the admins already have some mechanism to detect faulty nodes and then let it drain and reboot. However, if you are unlucky and it is your job currently hanging there, nothing is done about that. It needs manual inspection by the user and manual cancelation of the job (or waiting until the timelimit is hit). This is of course very suboptimal, and we are thinking about solutions. It was suggested that such logic could also be implemented as part of the Sisyphus manager.
Sisyphus already detects potential hanging jobs by checking whether the log has not been updated recently, and then prints:
[2024-01-28 22:12:30,257] WARNING: Job marked as running but logging file has not been updated: Job<work/i6_core/returnn/forward/ReturnnForwardJobV2.R8gexaONIr8S> assume it is running
So, the question is, how to detect whether it is really hanging and should be canceled and restarted.
And then the job should not go into error state, also not retry_error, but just restart it? Or maybe, if it can not be resumed, it should also automatically be cleared first?
And maybe Sisyphus should also keep a temporary local list of excluded nodes and temporarily add the node there (e.g. for 2h or so)?
Here some possible hang detections. All of them would involve logging on to the node via SSH.
-
Check if nvidia-smi returns an error.
(Maybe some unrelated GPU is faulty, but anyway, if your proc hangs + this is the case, I think it's ok to cancel the job.)
(This covers already a lot of cases I had, but not all.)
-
Check whether python -c "import torch; torch.cuda.init()" hangs.
-
Check py-spy dump -p <pid>, whether that hangs (given some timeout, maybe 10sec).
(One problem: Which PID actually? All the (deep) childs of slurm_script? Only the direct childs would actually not cover my RETURNN training setups with the option use_train_proc_manager, where only the sub sub proc hangs.)
(Alternative to py-spy: Maybe just strace, or gdb, or sth else? strace output looks like strace: Process 81401 attached and then nothing more comes. py-spy is also only for Python, so not generic.)
-
Check dmesg for some common errors? But what exactly?
Currently the admins already have some mechanism to detect faulty nodes and then let it drain and reboot. However, if you are unlucky and it is your job currently hanging there, nothing is done about that. It needs manual inspection by the user and manual cancelation of the job (or waiting until the timelimit is hit). This is of course very suboptimal, and we are thinking about solutions. It was suggested that such logic could also be implemented as part of the Sisyphus manager.
Sisyphus already detects potential hanging jobs by checking whether the log has not been updated recently, and then prints:
So, the question is, how to detect whether it is really hanging and should be canceled and restarted.
And then the job should not go into error state, also not retry_error, but just restart it? Or maybe, if it can not be resumed, it should also automatically be cleared first?
And maybe Sisyphus should also keep a temporary local list of excluded nodes and temporarily add the node there (e.g. for 2h or so)?
Here some possible hang detections. All of them would involve logging on to the node via SSH.
Check if
nvidia-smireturns an error.(Maybe some unrelated GPU is faulty, but anyway, if your proc hangs + this is the case, I think it's ok to cancel the job.)
(This covers already a lot of cases I had, but not all.)
Check whether
python -c "import torch; torch.cuda.init()"hangs.Check
py-spy dump -p <pid>, whether that hangs (given some timeout, maybe 10sec).(One problem: Which PID actually? All the (deep) childs of slurm_script? Only the direct childs would actually not cover my RETURNN training setups with the option
use_train_proc_manager, where only the sub sub proc hangs.)(Alternative to py-spy: Maybe just strace, or gdb, or sth else? strace output looks like
strace: Process 81401 attachedand then nothing more comes. py-spy is also only for Python, so not generic.)Check
dmesgfor some common errors? But what exactly?