Should running of optional expired tasks reset their try count to 0?

[ColemanTom]

Say you have an optional task which has run a few times and failed. The status is modified to expired. Someone then reruns it. Should the try number be reset to 1, or should it pick up from where it last tried? At the moment, it will reset to 1.

$ grep INIT_TIME ~/cylc-run/basic-workflow/log/job/20250123T1800Z/*/0?/job.status
/home/username/cylc-run/basic-workflow/log/job/20250123T1800Z/a_0/01/job.status:CYLC_JOB_INIT_TIME=2025-06-30T01:51:36Z
/home/username/cylc-run/basic-workflow/log/job/20250123T1800Z/a_0/02/job.status:CYLC_JOB_INIT_TIME=2025-06-30T01:51:42Z
/home/username/cylc-run/basic-workflow/log/job/20250123T1800Z/a_0/03/job.status:CYLC_JOB_INIT_TIME=2025-06-30T01:51:47Z
# expire "a_0" and then retrigger it
$ grep INIT_TIME ~/cylc-run/basic-workflow/log/job/20250123T1800Z/*/0?/job.status
/home/username/cylc-run/basic-workflow/log/job/20250123T1800Z/a_0/01/job.status:CYLC_JOB_INIT_TIME=2025-06-30T01:59:46Z
/home/username/cylc-run/basic-workflow/log/job/20250123T1800Z/a_0/02/job.status:CYLC_JOB_INIT_TIME=2025-06-30T01:59:52Z
/home/username/cylc-run/basic-workflow/log/job/20250123T1800Z/a_0/03/job.status:CYLC_JOB_INIT_TIME=2025-06-30T01:59:57Z

For reference, the workflow I ran to test this was using Cylc 8.3.6 and 8.4.3. The flow.cylc file was

[scheduler]
    allow implicit tasks = True
[scheduling]
    initial cycle point = 20250123T18
    runahead limit = P2

    [[graph]]
        T06, T18 = """
            a_0:failed? => b_0
            a_0[-PT12H]? | b_0[-PT12H] => a_0?

"""
[runtime]
    [[root]]
        script = false
        execution retry delays = PT1S, PT1S
        execution time limit = PT60S
    [[b_0]]
        platform = dm
        [[[directives]]]
            -l select=1:ncpus=1:mem=2G

I found this issue only happens for a_0 and not for b_0 if you expire and retry it, b_0 lauched try 04 instead of 01 again, so it appears to be linked to expiring optional tasks only.

Not shown here, but the platform of the task does not matter, only whether it has the optional+expired combination.

[ColemanTom]

Just checking that this one has been seen @oliver-sanders just because it does not have a label attached to it.

[oliver-sanders]

We're struggling for time right now so haven't prioritised this yet. I suspect the answer to your question is most likely "no it shouldn't reset".

[hjoliver]

Agreed, it shouldn't reset. I suspect the fact that it (evidently) does is accidental.

[oliver-sanders]

Replicated with this example:

[scheduling]
    [[graph]]
        R1 = """
            a?
            a:succeeded? => b
            a:failed? => broke
        """

[runtime]
    [[a]]
        script = false
        execution retry delays = PT0S
    [[b]]
    [[broke]]
        script = """
            cylc set "$CYLC_WORKFLOW_ID//1/a" -o expired
            sleep 2
            cylc trigger "$CYLC_WORKFLOW_ID//1/a"
        """

$ cylc vip -n my-test -N
$ for sub in $(seq 1 4); do echo my-test//1/a/0$sub; cylc cat my-test//1/a/0$sub -f j | grep TRY_NUM; done
tmp.XWgLbwGzAJ//1/a/01
    export CYLC_TASK_TRY_NUMBER=1
tmp.XWgLbwGzAJ//1/a/02
    export CYLC_TASK_TRY_NUMBER=2
tmp.XWgLbwGzAJ//1/a/03
File not found: /home/users/oliver.sanders/cylc-run/tmp.XWgLbwGzAJ/run3/log/job/1/a/03/job
tmp.XWgLbwGzAJ//1/a/04
File not found: /home/users/oliver.sanders/cylc-run/tmp.XWgLbwGzAJ/run3/log/job/1/a/04/job

When the task is expired, it leaves the pool. When it is subsequently triggered, it re-enters the pool, but the try and submission number are reset (because it's a new TaskProxy object).

This is "safe" in that Cylc will delete job output files from prior submissions when the first submission of a job is being prepared, but it is unhelpful in this scenario.

The issue is conceptually similar to #6143 in that it requires us to retrieve information about any prior instances of the task at spawn time.

Suggest resolving the two together.