deveshThe problem
The platform ran in a single AWS region. Disaster-recovery "plan" was a document nobody had tested: restore from backups, rebuild infrastructure by hand, estimated recovery time around 4 hours — and that estimate assumed everything went well. For a service with contractual uptime commitments, a regional outage was an existential risk sitting quietly on the books.
The approach
Define RTO/RPO before touching infrastructure. With stakeholders we agreed on a recovery time objective in minutes, not hours, and a recovery point objective near zero for transactional data. Those two numbers drove every technical decision that followed.
Data layer first. The primary database moved to Aurora Global Database, giving a continuously replicated read replica in the secondary region with sub-second replication lag and managed regional failover.
Everything reproducible. The entire secondary-region stack — networking, compute, secrets, DNS records — was expressed in the same Terraform modules as the primary, parameterised by region. The DR environment is not a special snowflake; it is the same code with different variables.
Health-checked DNS failover. Route53 health checks watch the primary region's public endpoints; failover routing shifts traffic to the secondary region automatically when checks fail, with TTLs tuned low enough to matter.
Game days. The part most DR projects skip: we deliberately failed the primary region in scheduled exercises, timed the recovery end-to-end, found the gaps (an IAM role that existed only in one region, a hardcoded regional endpoint), fixed them, and re-ran until the drill was boring.
The outcome
- RTO cut from ~4 hours to 8 minutes, measured in a live game day, not
estimated on paper.
- Near-zero RPO for transactional data via Aurora global replication.
- Failover is push-button (and partially automatic), documented in a runbook a
first-week on-call engineer can execute.
- DR moved from an audit checkbox to a rehearsed capability.