inconsistent resubmission behavior for spot fail

[moe1619]

Hello!

Thank you for creating this wonderful plugin. The spot retry behavior seems a bit inconsistent. Sometimes, a spot interuption is followed seamlessly by a resubmission. Othertimes, the spot failure seems to immediately propogate. Two images below show the different behavior.

successful

no attempt made to resubmit

Any ideas?

Thanks,
Josh

[mlin]

@moe1619 The default cfg allows up to 2 retries (3 total attempts) of a task. Is it possible that's being exhausted in these cases? You could search the log for 3 prior instances of "interrupted task will be retried" for the same call name.

If so, you have a few options:

1. Edit the WDL to set runtime.preemptible to a higher value like 5
2. Override that default cfg, usually the easiest way is to set e.g. MINIWDL__TASK_RUNTIME__DEFAULTS='{"docker":"ubuntu:20.04","preemptible":5}' in the environment to miniwdl-aws-submit.
3. If you're using miniwdl-aws-terraform, it can be configured to make a last-ditch attempt using an on-demand queue after so many spot interruptions (see enable_task_fallback).

[mlin]

Miniwdl itself could provide a better log message when the allowed attempts are exhausted.

[moe1619]

Thanks for your reply. I'll look into it.

[moe1619]

Thank you again for this tool. Its been really fantastic.

I am still doing some more confirmation but I believe the spot resubmission error occurs when the preemptible value is set to 0 AND the task queue is connected to a spot compute environment AND the spot EC2 is terminated. (The behavior seemed random to me because the preemptible value is set dynamically which I had forgotten about)

Ideally, setting preemptible to 0 would prompt using the fallback queue immediately and ignoring the task queue entirely. In the current code, the task is submitted to the task queue (which may or may not be spot) even if the preemptible is set to 0 (I think).