Designing Systems That Survive a Region Outage

Cloud regions are remarkably reliable, and they still fail. When they do, the outage often comes from a shared dependency rather than a whole data center going dark: a DNS problem, an identity service, a control plane that won't accept changes. Surviving that kind of event is an architecture decision made months earlier, and it starts with a clear statement of what the business can tolerate.

Start from RTO and RPO

The recovery time objective (RTO) is how long a service can be unavailable. The recovery point objective (RPO) is how much recent data you can afford to lose. Both should be set per service, by the business, with the cost in front of them, because moving from hours to minutes typically multiplies infrastructure spend and operational complexity.

• Tier 0, revenue-critical paths: RTO in minutes, RPO near zero.
• Tier 1, important internal and customer services: RTO within an hour or two, RPO of minutes.
• Tier 2, everything else: RTO within a day, restored from backups.

The four patterns and their trade-offs

• Backup and restore: the cheapest option. Infrastructure is recreated from code and data from backups. RTO is measured in hours.
• Pilot light: data is replicated continuously and core components exist in the second region but are scaled to zero. Recovery means scaling up, usually within tens of minutes.
• Warm standby: a smaller, fully working copy runs in the second region and is scaled up during failover. RTO of minutes.
• Active-active: both regions serve production traffic all the time. Failover is mostly a traffic shift, but every part of the system must cope with running in two places at once.

The data problem behind every failover

Stateless services are easy to run in two regions. Data is where the hard decisions live. Synchronous replication across regions guarantees no lost writes, but every write now pays the round trip between regions, which is bounded by physics and often adds tens of milliseconds. Asynchronous replication keeps writes fast but means RPO is never truly zero; anything not yet replicated when the region fails is lost or must be reconciled later.

Active-active writes add conflict resolution to the list. Last-writer-wins is simple and silently discards data. Conflict-free replicated data types work well for counters and sets but not for every business object. Globally distributed databases such as Spanner, CockroachDB, Aurora Global Database and DynamoDB global tables each make different trade-offs between consistency, latency and cost, and those trade-offs deserve a careful read before you commit. Whatever you choose, make write operations idempotent so that retries during a failover cannot charge a customer twice.

Limit the blast radius with cells

Multi-region protects against losing a region. Cell-based architecture protects against your own bad deploys and poison requests, which are far more common. The system is split into independent cells, each serving a subset of customers with its own copy of the stack. A failure, or a bad release rolled out cell by cell, affects only the customers in that cell. Shuffle sharding takes the idea further by giving each customer a different combination of resources, so one noisy customer is unlikely to take down another.

Moving traffic when it matters

• Design for static stability: failover should not depend on creating resources or calling control planes in the failed region. Capacity in the standby should already exist.
• Do not rely on DNS TTLs alone. Some clients and resolvers cache longer than you ask. Global load balancers or anycast give faster, more predictable shifts.
• Health checks should test real dependencies, but avoid a single shared dependency failing every health check at once and draining both regions.
• Decide in advance who has the authority to fail over, and how manual that decision is. Automatic failover on a flapping signal can cause more damage than the original outage.

Prove it before you need it

Run game days where you fail over a production service on purpose, during working hours, with the people who would be on call. Use fault injection tools such as AWS Fault Injection Service, Azure Chaos Studio or Chaos Mesh to rehearse smaller failures continuously. Record the actual recovery time and data loss each time and compare them with the targets you promised. The gap between the two is your resilience roadmap.