Best Practices for Handling Task Cancellation in Workflows

[pcabaleiro]

Hello,

I'm currently working with a workflow that includes several time-consuming tasks. I've observed that when I cancel a workflow through the API (for instance, to modify an input parameter), tasks that have already commenced are marked as CANCELED in the Conductor's service. However, these tasks continue to run on the worker side until they either complete or fail.

I'm seeking clarity on a couple of points:

• Is this continuation of tasks post-cancellation an expected behavior in such workflows?
• Are there established strategies or best practices within the community for effectively halting workers from completing a task once it has been marked as canceled?

Any insights or guidance on this would be greatly appreciated.

Thank you!

[ghkdwlgns612]

Hello @pcabaleiro

I also had a hard time working on time-consuming. I don't know if it helps, but I'll share the solutions we're using.
Only applicable to task with logic waiting to Thread.sleep() after Non-blocking.

public TaskResult execute(Task task) {

    .....
    ExecutorService executorService = Executors.newSingleThreadExecutor();
    executorService.execute(() -> executeLogic(task, taskResult));
    
    try {
        checkTaskStatusPerSeconds(task, executorService);
    } catch (Exception ignored) {
    }

}
....

private void checkTaskStatusPerSeconds(Task task, ExecutorService executorService) throws InterruptedException {
    while (true) {
        Task taskDetails = taskClient.getTaskDetails(task.getTaskId());
        if (taskDetails.getStatus().equals(Task.Status.CANCELED)
                || taskDetails.getStatus().equals(Task.Status.COMPLETED)
                || taskDetails.getStatus().equals(Task.Status.FAILED)
                || taskDetails.getStatus().equals(Task.Status.FAILED_WITH_TERMINAL_ERROR)) {
            log.info("Status observer closed");
            executorService.shutdownNow();
            break;
        }
        Thread.sleep(1000L);
    }
}

If you try to kill workflow using workflowClient.terminateWorkflow, In the conductor-server, the task will be in CANCLE or FAIL state.
It can be read in Polling form with the checkTaskStatusPerSeconds() in Worker. Task results run CANCLE or FAILED side executorService.shutdownNow().

[ghkdwlgns612]

Below is a plan provided by the document. But I don't work...