Runs that are in the `WAITING` state don't pick up latest changes

[LudovicoRighi]

Updating the job config (i.e. the command) does not have any effect on the corresponding runs that are already in the waiting state (but only on those that will be scheduled after)

What do you think about cancelling and rescheduling waiting runs when a job's program changes?

cc @gusostow

[alexhsamuel]

I assume you mean "when a job's conditions change".

See the discussion in #336. I don't think there are clear semantics to do this.

For instance, what if the schedule also changed? Do we reschedule the waiting run even if this run would not have been scheduled? But then what if the conditions haven't changed? We don't cancel such a run.

I think the only semantics that are straightforward to articulate and clear in all cases are that a run is fixed at the time its scheduled time elapses, i.e. when it enters waiting. After that, it no longer changes.

[LudovicoRighi]

For instance, what if the schedule also changed? Do we reschedule the waiting run even if this run would not have been scheduled? But then what if the conditions haven't changed? We don't cancel such a run.

Ok, right. I agree that we should definitely not consider the changes to the schedules or conditions, as there isn't clear semantics, as you highlighted.

I think the only semantics that are straightforward to articulate and clear in all cases are that a run is fixed at the time its scheduled time elapses, i.e. when it enters waiting. After that, it no longer changes.

But how about making an exception to this ^ for the command/argv (i.e. pick up the latest command/argv)?
I think that if people have changed the command/argv, they'd always want to start using it from the very next run; can you think of any scenario in which this is not true?

[alexhsamuel]

Any changes to runs past the scheduled state opens up the huge set of ambiguities and corner cases we've previously discussed. We should avoid these.