Skip to content

Commit 7794a2e

Browse files
lukeknepclaude
andcommitted
Address review feedback on conflict resolution and Workflow Id uniqueness
- Fix the mangled sentence in the replication-backlog bullet (temporal-nick). - Move the permanent-data-loss caveat into a caution box (brianmacdonald-temporal). - Correct the fork description: conflict resolution keeps one Workflow Execution Open, and the competing Execution in the previously active region becomes a zombie Execution that is terminated once replication reports the conflict (temporal-nick). - Add SignalWithStartWorkflowExecution and UpdateWorkflowExecution to the durability boundary (yux0). - Disambiguate the uniqueness guarantee from the Workflow Id Reuse Policy and Workflow Id Conflict Policy, linking each (yux0). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent f58c302 commit 7794a2e

1 file changed

Lines changed: 32 additions & 20 deletions

File tree

  • docs/cloud/high-availability/failovers

docs/cloud/high-availability/failovers/index.mdx

Lines changed: 32 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -154,15 +154,21 @@ remains active.
154154

155155
Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be
156156
reflected in the replica due to <ToolTipTerm term="replication lag" />, particularly during failovers. In the event of a
157-
non-graceful failover, replication lag causes temporary setback in Workflow progress. At the moment of non-graceful
157+
non-graceful failover, replication lag causes a temporary setback in Workflow progress. At the moment of non-graceful
158158
failover:
159159

160160
- Operations that had already replicated remain durable in the replica.
161161
- Operations that had not yet replicated (i.e., that are still in the replication backlog) are reconciled when the
162-
region recovers. according to the Conflict Resolution process. - Note that Conflict Resolution can only recover data
163-
from a functioning Temporal Server. If the active server never recovers, the Workflow API calls that fall within the 1
164-
minute RPO may be permanently lost. Such a case would require the permanent loss of multiple cloud Availability Zones
165-
and has never happened in the history of Temporal Cloud.
162+
region recovers, according to the conflict resolution process.
163+
164+
:::caution Conflict resolution requires a recoverable region
165+
166+
Conflict resolution can only recover data from a functioning Temporal Service. If the previously active region never
167+
recovers, Workflow API calls that fall within the [RPO](/cloud/rpo-rto) — under one minute — may be permanently lost.
168+
Such a case would require the permanent loss of multiple cloud Availability Zones and has never happened in the history
169+
of Temporal Cloud.
170+
171+
:::
166172

167173
In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching
168174
regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero.
@@ -172,26 +178,32 @@ retry policy's [maximum attempts](https://docs.temporal.io/retry-policies#maximu
172178
Availability Namespaces provide _at-least-once_ semantics for execution of Activities. Completed Activities _may_ be
173179
re-dispatched in a newly active Namespace, leading to repeated executions.
174180

175-
The same durability boundary applies to Workflow starts and Signals: a `StartWorkflowExecution` or
176-
`SignalWorkflowExecution` call that returns success is durably committed in the active region, and replicated
177-
asynchronously to the replica.
181+
The same durability boundary applies to Workflow starts, Signals, and Updates: a `StartWorkflowExecution`,
182+
`SignalWorkflowExecution`, `SignalWithStartWorkflowExecution`, or `UpdateWorkflowExecution` call that returns success is
183+
durably committed in the active region, and replicated asynchronously to the replica.
178184

179185
### How Workflow Id uniqueness is preserved after a forced failover
180186

181-
The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid) — at most one Open Workflow Execution per
182-
Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across a failover.
183-
The guarantee limits how many Executions are _Open_ at the same time, not how many
184-
[Run Ids](/workflow-execution/workflowid-runid#run-id) a Workflow Id accumulates over its lifetime. That distinction is
185-
what lets conflict resolution reconcile a divergence without ever running the same Workflow Id twice concurrently.
187+
The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid#workflow-id) — at most one Open Workflow
188+
Execution per Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across
189+
a failover. This guarantee limits how many Executions are _Open_ at the same time; reuse of a Workflow Id after an
190+
Execution Closes is governed separately by the
191+
[Workflow Id Reuse Policy](/workflow-execution/workflowid-runid#workflow-id-reuse-policy), and a start request that
192+
collides with an already-Open Execution is governed by the
193+
[Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy). Because the guarantee
194+
constrains only concurrency, and not how many [Run Ids](/workflow-execution/workflowid-runid#run-id) a Workflow Id
195+
accumulates over its lifetime, conflict resolution can reconcile a divergence without ever running the same Workflow Id
196+
twice concurrently.
186197

187198
1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event
188199
History to the replica.
189200
2. **Failover with divergence.** In a forced failover when replication lag is present, both regions can independently
190-
append events to the same Workflow Execution. When the regions reconnect, their Event Histories have diverged for
191-
that Workflow Id.
192-
3. **Fork instead of merge.** Temporal Cloud does not interleave the divergent histories. Events from the previously
201+
append events under the same Workflow Id. When the regions reconnect, their Event Histories have diverged for that
202+
Workflow Id.
203+
3. **One Execution stays Open.** Temporal Cloud does not interleave the divergent histories. Events from the previously
193204
active Namespace that arrive after the failover cannot be directly applied, so Temporal Cloud forks the Event History
194-
and creates a new branch history, each branch identified by its own Run Id. Its
195-
<ToolTipTerm term="conflict resolution" /> process keeps one branch as the Open Execution and supersedes the other,
196-
leaving exactly one Open Workflow Execution per Workflow Id. The Temporal Service ensures the resulting Event
197-
Histories remain valid and replayable by SDKs.
205+
into a new branch. Its <ToolTipTerm term="conflict resolution" /> process then keeps a single Workflow Execution
206+
Open. The competing Execution in the previously active region becomes a
207+
[zombie Workflow Execution](/temporal-service/multi-cluster-replication#zombie-workflows) — an Execution that region
208+
can no longer mutate on its own — and is terminated there once replication informs it of the competing Workflow Id.
209+
The Temporal Service ensures the resulting Event Histories remain valid and replayable by SDKs.

0 commit comments

Comments
 (0)