Issue #12: AWS Was Down, How About Your Application?

In the last few days all I could see on LinkedIn was AWS, AWS, AWS… One region goes down, half the world goes down. Everybody talks about what caused it; very few talk about how not to be affected by it. “This incident only shows how reliant on AWS and the cloud companies are!” is what I mostly read. But in between the lines, I’d risk saying that what this incident surfaced was not overreliance on AWS, but rather how unprepared most companies are for this kind of situation. A cloud region is bound to eventually go down, so what matters is how well-prepared you and your systems are to deal with the disruption. Disaster recovery is not straightforward, yes, and the consequence is that it ends up being pushed back as new features (ooh, the new features) take priority. However, those who had an effective, clear, and tested disaster recovery plan were able to respond to the incident systematically and had their systems recover much faster than the AWS region itself.

So in this article, we’re gonna dive deeper into the four main disaster recovery strategies, how they differ from each other, what are the pros and cons of each, the trade-offs between them, and when to use each. It’s a lot of ground to cover, so let’s get started!

Before we dive into the strategies, we need to understand the two fundamental trade-offs in disaster recovery. Everything else is built on these two concepts:

The golden rule is this: the lower (and better) your RTO and RPO, the higher the cost and complexity. Your job is to find the right balance for each part of your system. Here's a simple way to visualize it on a timeline:

How much data loss and downtime you are willing to tolerate really depends on which system you are evaluating. Your internal analytics dashboard might have an RTO of 4 hours and an RPO of 1 hour, meaning it's okay if it's down for a bit and you lose a small amount of recent data. But your mission-critical payment processing system would probably need an RTO of less than 1 minute and an RPO of a few seconds, because every moment of downtime and every lost transaction costs real money! Customer-facing systems normally have lower RPO and RTO, since every second of downtime impacts your customers’ experience and can have a negative impact on your image and future revenue.

With RTO and RPO in mind, let's walk through the four common DR strategies. I love using analogies to make these stick, so bear with me and my little examples 🤓

This is the most common and straightforward approach. In a few words, what you do here is you regularly take snapshots of your data (like a database backup or a disk image) and store them somewhere safe and cheap, like Amazon S3 Glacier or Google Coldline Storage. If disaster strikes, you provision new infrastructure and "restore" from that backup.

It’s like keeping a copy of your important documents in a safe deposit box across town. It's very secure, but should you actually need them, it takes time to go to the bank, get the documents, and bring them back.

As already mentioned, each strategy has its pros and cons. For the backup and restore, we can think of the following:

When to use: Because of its high RTO and RPO, this strategy is not recommended for production systems. It’s perfect for development environments, non-critical services, or systems where you can tolerate significant downtime and some data loss, but that’s as far as I’d go in terms of using it.

This is where things get more interesting. With the Pilot Light strategy, you replicate your data in real time to a DR region. The core infrastructure needed to run the service (like the database) is kept running, but the main application servers are turned off. In a disaster, you "turn on" the servers, scale them up, and point your traffic to the new region. Sounds a bit too abstract? For me too 💥 So let’s dive a bit more into how it would work by breaking it down by layer:

When a disaster strikes, the "ignition" process is triggered:

In terms of pros and cons for the Pilot Light, we can think of the following:

When to use it: This strategy provides a great middle ground for important but not mission-critical applications where a few minutes to an hour of downtime is acceptable. Since the restoration process still requires some manual work (scaling resources, redirecting traffic), there is still some noticeable downtime to end users, so this is not suited for highly critical production systems.

A Warm Standby is a scaled-down but fully functional version of your production environment running in the DR region. Data is replicated in real-time, and the servers are on and running - they just aren't at full production scale. During a failover, all you have to do is switch traffic over and quickly scale up the resources to handle the full load.

Think of it as a backup generator for a shopping mall (I was about to write for your home, but really… who has a backup generator at home??). The generator is always running and ready, so when the power goes out, it kicks in automatically and can run the essentials (but not everything). The analogy fails a bit to capture the fact that, while you can’t scale a generator, you can scale your cloud resources to meet the production requirements.

In terms of pros and cons:

When to use it: This is probably where your core business applications would fall. It’s ideal for systems where downtime is measured in thousands of dollars per minute, but some minimal outage is still acceptable.

This is the gold standard of availability. In this setup, you have two or more fully independent, production-scale environments running in different regions. A global load balancer distributes traffic between them. If one region fails, the other is already handling a portion of the load and simply takes over the rest seamlessly.

Think of a retail chain with two fully staffed stores in the same city. If one has to close for an emergency, customers are already used to going to the other, and there’s no interruption in service.

Here are some pros and cons of the Multi-Site Active-Active strategy:

When to use it: Because of its complexity, this strategy is probably overkill for most systems. That being said, it can be the right choice for absolute mission-critical services that cannot go down, like payment processing systems or authentication services.

As you can see, disaster recovery isn't a one-size-fits-all problem. It’s a spectrum, ranging from the cheap-and-slow "safe deposit box" to the expensive-and-instant "twin city."

The goal isn't to use Active-Active for everything. The goal is to look at each component of your system, understand its unique RTO and RPO requirements, and choose the most appropriate and cost-effective strategy for the job.

I hope this helps demystify the topic a bit! It’s a fascinating area where smart engineering can provide huge business value.

What DR strategies are you using today? I'd love to hear about your experiences or any disaster recovery stories you have to share. Just hit reply and let me know!

Thanks for reading this edition of Ctrl+Alt+Deploy. Found these insights valuable? Share this newsletter with fellow developers and let me know which story resonated with you most!

Until next time, keep coding and stay curious! 💻✨

💡 Curated with ❤️ for the developer community