diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 50b0018409..e84db9f887 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -31,9 +31,10 @@ failover, Temporal automatically fails back to the original region once it is he ## Automatic failover -Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. -These automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage and complete a failover in minutes from when the outage began, -according to the stated [Recovery Time Objective (RTO)](/cloud/rpo-rto). +Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. These +automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage +and complete a failover in minutes from when the outage began, according to the stated +[Recovery Time Objective (RTO)](/cloud/rpo-rto). After an automatic failover, the Namespace will have a replica in its original region. Once the original region is healthy again, Temporal Cloud automatically performs a [failback](/cloud/high-availability/failovers/manage#failbacks), @@ -64,8 +65,8 @@ regional outage. :::info -The following list gives a general idea of the conditions that trigger an automatic failover. This is not an -exhaustive list, and it may change over time. +The following list gives a general idea of the conditions that trigger an automatic failover. This is not an exhaustive +list, and it may change over time. ::: @@ -80,22 +81,23 @@ exhaustive list, and it may change over time. You can also [manually trigger a failover](/cloud/high-availability/failovers/manage#trigger-failover) based on your own monitoring or for failover testing. -Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover -is warranted are: +Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover (i.e., +a failover triggered by a user) is warranted are: - **Testing failover or migrating to a new region.** A manual failover is the standard way to exercise your failover process with your Clients and Workers, or to move a Namespace to a different region. - **An outage that affects only your systems.** If an outage is contained to your application, Workers, or other infrastructure, and Temporal Cloud is not affected, Temporal will not initiate a failover on your behalf. Detect the outage with your own monitoring and trigger a failover yourself. -- **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can - trigger a failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first - takes effect, and the later one is a no-op. A user-triggered failover does not conflict with Temporal's automatic - failover. +- **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can trigger a + failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first takes effect, + and the later one is a no-op. A manual failover does not conflict with Temporal's automatic failover. :::note Same-region Replication -Manual failovers apply only to Multi-region and Multi-cloud Replication. A [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between cells and cannot be failed over manually or have its automatic failovers disabled. +Manual failovers apply only to Multi-region and Multi-cloud Replication. A +[Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between +cells and cannot be failed over manually or have its automatic failovers disabled. ::: @@ -127,9 +129,15 @@ The failover process is the same whether it is triggered automatically by Tempor ## Post-failover events {/* #post-failover-events */} -After any failover, whether triggered by you or by Temporal, an event appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud [notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs. +After any failover, whether triggered by you or by Temporal, an event appears in both the +[Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The +audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud +[notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs. -After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover, the Namespace stays in the replica region until a user triggers another failover. See [failback options](/cloud/high-availability/failovers/manage#failbacks) for details. +After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless +you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover, +the Namespace stays in the replica region until a user triggers another failover. See +[failback options](/cloud/high-availability/failovers/manage#failbacks) for details. ## Split-brain scenario @@ -146,15 +154,56 @@ remains active. Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be reflected in the replica due to , particularly during failovers. In the event of a -non-graceful failover, replication lag may cause a temporary setback in Workflow progress. +non-graceful failover, replication lag causes a temporary setback in Workflow progress. At the moment of non-graceful +failover: + +- Operations that had already replicated remain durable in the replica. +- Operations that had not yet replicated (i.e., that are still in the replication backlog) are reconciled when the + region recovers, according to the conflict resolution process. + +:::caution Conflict resolution requires a recoverable region + +Conflict resolution can only recover data from a functioning Temporal Service. If the previously active region never +recovers, Workflow API calls that fall within the [RPO](/cloud/rpo-rto) — under one minute — may be permanently lost. +Such a case would require the permanent loss of multiple cloud Availability Zones and has never happened in the history +of Temporal Cloud. + +::: + +In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching +regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero. Namespaces that are not replicated can be configured to provide _at-most-once_ semantics for Activity execution when a retry policy's [maximum attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0. High Availability Namespaces provide _at-least-once_ semantics for execution of Activities. Completed Activities _may_ be re-dispatched in a newly active Namespace, leading to repeated executions. -When a Workflow Execution is updated in a newly active replica following a failover, events from the previously active -Namespace that arrive after the failover cannot be directly applied. At this point, Temporal Cloud has forked the Event -History. - -After failover, Temporal Cloud creates a new branch history for execution and begins its process. The Temporal Service ensures that Event Histories remain valid and are replayable by SDKs post-failover or after conflict resolution. +The same durability boundary applies to Workflow starts, Signals, and Updates: a `StartWorkflowExecution`, +`SignalWorkflowExecution`, `SignalWithStartWorkflowExecution`, or `UpdateWorkflowExecution` call that returns success is +durably committed in the active region, and replicated asynchronously to the replica. + +### How Workflow Id uniqueness is preserved after a forced failover + +The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid#workflow-id) — at most one Open Workflow +Execution per Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across +a failover. This guarantee limits how many Executions are _Open_ at the same time; reuse of a Workflow Id after an +Execution Closes is governed separately by the +[Workflow Id Reuse Policy](/workflow-execution/workflowid-runid#workflow-id-reuse-policy), and a start request that +collides with an already-Open Execution is governed by the +[Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy). Because the guarantee +constrains only concurrency, and not how many [Run Ids](/workflow-execution/workflowid-runid#run-id) a Workflow Id +accumulates over its lifetime, conflict resolution can reconcile a divergence without ever running the same Workflow Id +twice concurrently. + +1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event + History to the replica. +2. **Failover with divergence.** In a forced failover when replication lag is present, both regions can independently + append events under the same Workflow Id. When the regions reconnect, their Event Histories have diverged for that + Workflow Id. +3. **One Execution stays Open.** Temporal Cloud does not interleave the divergent histories. Events from the previously + active Namespace that arrive after the failover cannot be directly applied, so Temporal Cloud forks the Event History + into a new branch. Its process then keeps a single Workflow Execution + Open. The competing Execution in the previously active region becomes a + [zombie Workflow Execution](/temporal-service/multi-cluster-replication#zombie-workflows) — an Execution that region + can no longer mutate on its own — and is terminated there once replication informs it of the competing Workflow Id. + The Temporal Service ensures the resulting Event Histories remain valid and replayable by SDKs. diff --git a/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx b/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx index 23bf90699e..050366a4fc 100644 --- a/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx +++ b/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx @@ -27,6 +27,13 @@ In some cases, such as when running the same Workflow at recurring intervals usi In this case, all runs will have the same Workflow ID. However, each run will have a unique system-generated [Run ID](#run-id). +:::note Workflow Id uniqueness and High Availability + +This uniqueness constraint is enforced within the active [Namespace](/namespaces). +On a [High Availability](/cloud/high-availability) Namespace, replication is asynchronous, and a failover preserves the constraint through Temporal Cloud's [conflict resolution](/cloud/high-availability/failovers#conflict-resolution) process. + +::: + ## What is a Run Id? {/* #run-id */} A Run Id is a globally unique, platform-level identifier for a [Workflow Execution](/workflow-execution).