Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 69 additions & 20 deletions docs/cloud/high-availability/failovers/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,10 @@ failover, Temporal automatically fails back to the original region once it is he

## Automatic failover

Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability.
These automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage and complete a failover in minutes from when the outage began,
according to the stated [Recovery Time Objective (RTO)](/cloud/rpo-rto).
Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. These
automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage
and complete a failover in minutes from when the outage began, according to the stated
[Recovery Time Objective (RTO)](/cloud/rpo-rto).

After an automatic failover, the Namespace will have a replica in its original region. Once the original region is
healthy again, Temporal Cloud automatically performs a [failback](/cloud/high-availability/failovers/manage#failbacks),
Expand Down Expand Up @@ -64,8 +65,8 @@ regional outage.

:::info

The following list gives a general idea of the conditions that trigger an automatic failover. This is not an
exhaustive list, and it may change over time.
The following list gives a general idea of the conditions that trigger an automatic failover. This is not an exhaustive
list, and it may change over time.

:::

Expand All @@ -80,22 +81,23 @@ exhaustive list, and it may change over time.
You can also [manually trigger a failover](/cloud/high-availability/failovers/manage#trigger-failover) based on your own
monitoring or for failover testing.

Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover
is warranted are:
Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover (i.e.,
a failover triggered by a user) is warranted are:

- **Testing failover or migrating to a new region.** A manual failover is the standard way to exercise your failover
process with your Clients and Workers, or to move a Namespace to a different region.
- **An outage that affects only your systems.** If an outage is contained to your application, Workers, or other
infrastructure, and Temporal Cloud is not affected, Temporal will not initiate a failover on your behalf. Detect the
outage with your own monitoring and trigger a failover yourself.
- **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can
trigger a failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first
takes effect, and the later one is a no-op. A user-triggered failover does not conflict with Temporal's automatic
failover.
- **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can trigger a
failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first takes effect,
and the later one is a no-op. A manual failover does not conflict with Temporal's automatic failover.

:::note Same-region Replication

Manual failovers apply only to Multi-region and Multi-cloud Replication. A [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between cells and cannot be failed over manually or have its automatic failovers disabled.
Manual failovers apply only to Multi-region and Multi-cloud Replication. A
[Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between
cells and cannot be failed over manually or have its automatic failovers disabled.

:::

Expand Down Expand Up @@ -127,9 +129,15 @@ The failover process is the same whether it is triggered automatically by Tempor

## Post-failover events {/* #post-failover-events */}

After any failover, whether triggered by you or by Temporal, an event appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud [notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs.
After any failover, whether triggered by you or by Temporal, an event appears in both the
[Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The
audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud
[notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs.

After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover, the Namespace stays in the replica region until a user triggers another failover. See [failback options](/cloud/high-availability/failovers/manage#failbacks) for details.
After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless
you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover,
the Namespace stays in the replica region until a user triggers another failover. See
[failback options](/cloud/high-availability/failovers/manage#failbacks) for details.

## Split-brain scenario

Expand All @@ -146,15 +154,56 @@ remains active.

Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be
reflected in the replica due to <ToolTipTerm term="replication lag" />, particularly during failovers. In the event of a
non-graceful failover, replication lag may cause a temporary setback in Workflow progress.
non-graceful failover, replication lag causes a temporary setback in Workflow progress. At the moment of non-graceful
failover:

- Operations that had already replicated remain durable in the replica.
- Operations that had not yet replicated (i.e., that are still in the replication backlog) are reconciled when the
region recovers, according to the conflict resolution process.

:::caution Conflict resolution requires a recoverable region

Conflict resolution can only recover data from a functioning Temporal Service. If the previously active region never
recovers, Workflow API calls that fall within the [RPO](/cloud/rpo-rto) — under one minute — may be permanently lost.
Such a case would require the permanent loss of multiple cloud Availability Zones and has never happened in the history
of Temporal Cloud.

:::

In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching
regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero.

Namespaces that are not replicated can be configured to provide _at-most-once_ semantics for Activity execution when a
retry policy's [maximum attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0. High
Availability Namespaces provide _at-least-once_ semantics for execution of Activities. Completed Activities _may_ be
re-dispatched in a newly active Namespace, leading to repeated executions.

When a Workflow Execution is updated in a newly active replica following a failover, events from the previously active
Namespace that arrive after the failover cannot be directly applied. At this point, Temporal Cloud has forked the Event
History.

After failover, Temporal Cloud creates a new branch history for execution and begins its <ToolTipTerm term="conflict resolution" /> process. The Temporal Service ensures that Event Histories remain valid and are replayable by SDKs post-failover or after conflict resolution.
The same durability boundary applies to Workflow starts, Signals, and Updates: a `StartWorkflowExecution`,
`SignalWorkflowExecution`, `SignalWithStartWorkflowExecution`, or `UpdateWorkflowExecution` call that returns success is
durably committed in the active region, and replicated asynchronously to the replica.

### How Workflow Id uniqueness is preserved after a forced failover

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@temporal-nick can you check for correctness?


The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid#workflow-id) — at most one Open Workflow
Execution per Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across
a failover. This guarantee limits how many Executions are _Open_ at the same time; reuse of a Workflow Id after an
Execution Closes is governed separately by the
[Workflow Id Reuse Policy](/workflow-execution/workflowid-runid#workflow-id-reuse-policy), and a start request that
collides with an already-Open Execution is governed by the
[Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy). Because the guarantee
constrains only concurrency, and not how many [Run Ids](/workflow-execution/workflowid-runid#run-id) a Workflow Id
accumulates over its lifetime, conflict resolution can reconcile a divergence without ever running the same Workflow Id
twice concurrently.

1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event
History to the replica.
2. **Failover with divergence.** In a forced failover when replication lag is present, both regions can independently
append events under the same Workflow Id. When the regions reconnect, their Event Histories have diverged for that
Workflow Id.
3. **One Execution stays Open.** Temporal Cloud does not interleave the divergent histories. Events from the previously
active Namespace that arrive after the failover cannot be directly applied, so Temporal Cloud forks the Event History
into a new branch. Its <ToolTipTerm term="conflict resolution" /> process then keeps a single Workflow Execution
Open. The competing Execution in the previously active region becomes a
[zombie Workflow Execution](/temporal-service/multi-cluster-replication#zombie-workflows) — an Execution that region
can no longer mutate on its own — and is terminated there once replication informs it of the competing Workflow Id.
The Temporal Service ensures the resulting Event Histories remain valid and replayable by SDKs.
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,13 @@ In some cases, such as when running the same Workflow at recurring intervals usi
In this case, all runs will have the same Workflow ID.
However, each run will have a unique system-generated [Run ID](#run-id).

:::note Workflow Id uniqueness and High Availability

This uniqueness constraint is enforced within the active [Namespace](/namespaces).
On a [High Availability](/cloud/high-availability) Namespace, replication is asynchronous, and a failover preserves the constraint through Temporal Cloud's [conflict resolution](/cloud/high-availability/failovers#conflict-resolution) process.

:::

## What is a Run Id? {/* #run-id */}

A Run Id is a globally unique, platform-level identifier for a [Workflow Execution](/workflow-execution).
Expand Down