Overriding `waiting` `max_time` in individual job conditions

[LudovicoRighi]

It would be nice to have the possibility to override waiting.max_time for specific conditions/dependencies, rather than having this as a global Apsis setting.

cc @alexhsamuel

[alexhsamuel]

This timeout is per run, not per condition.

[alexhsamuel]

@LudovicoRighi

It doesn't make sense to override the per-run timeout from an individual condition. Some options we can consider:

1. Override the per-run timeout in the job somewhere. There isn't an obvious place to attach this, as there are currently no general job configs. It could perhaps go in metadata, or we create a new section in the job config.

2. Replace the per-run max_waiting with a per-condition timeout, and allow individual conditions to override this. This means there is no consistent per-run timeout, though.

At first blush, I think the latter is better. You're presumably setting max_waiting to 24 hours or so, which is way too long for almost all your jobs, but still not long enough for a few of them. The whole notion of a global timeout across such varied jobs doesn't make too much sense.

[LudovicoRighi]

Agreed that the second option is the best one.

So we would basically be able to do something like the following, right?

condition:
  - type: dependency
    job_id: AAA
    max_waiting: 60 min

  - type: dependency
    job_id: BBB
    max_waiting: 5 min

  - type: dependency
    job_id: CCC

cc @gusostow