Pre-requisites
What happened? What did you expect to happen?
When a workflow uses a database-backed mutex or semaphore and the SERIALIZABLE transaction in TryAcquire hits a 40001 serialization conflict, the workflow node is permanently marked Error instead of being retried.
acquire logs the SQL error but returns only false, dropping the cause. tryAcquireImpl then wraps the bare failure as "bug: failed to acquire something that should have been checked: ". The retry loop in TryAcquire substring-matches for serialization/dependencies/deadlock/rollback — none of which appear
in the string so the loop falls through to the non-retryable branch on attempt 1.
Net effect: a transient PostgreSQL serializable conflict becomes a permanent workflow-node
error.
Version(s)
:latest
Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
impossible to reproduce on a single controller setup.
Logs from the workflow controller
Logs from in your workflow's wait container
Pre-requisites
:latestimage tag (i.e.quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on:latest. If not, I have explained why, in detail, in my description below.What happened? What did you expect to happen?
When a workflow uses a database-backed mutex or semaphore and the SERIALIZABLE transaction in TryAcquire hits a 40001 serialization conflict, the workflow node is permanently marked Error instead of being retried.
acquire logs the SQL error but returns only false, dropping the cause. tryAcquireImpl then wraps the bare failure as "bug: failed to acquire something that should have been checked: ". The retry loop in TryAcquire substring-matches for serialization/dependencies/deadlock/rollback — none of which appear
in the string so the loop falls through to the non-retryable branch on attempt 1.
Net effect: a transient PostgreSQL serializable conflict becomes a permanent workflow-node
error.
Version(s)
:latest
Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
impossible to reproduce on a single controller setup.Logs from the workflow controller
Logs from in your workflow's wait container