Systems Design

Get It Back Working First

Published

A red power switch glowing against a dark server-rack background

Photo: Illustration by David O'Brien

A common shape for a system design interview question is to describe a system that's failing under load, and ask how you'd scale it. Most candidates hear "failing" and "scale" in the same sentence and jump straight to the scaling part - sharding the database, adding a cache, putting a CDN in front of everything.

That's the direction the answer is going to develop in anyway, so the temptation is to rush to the target end state, but stronger candidates will first look to understand what the failure is and how it can be addressed. This is a great point to learn more about the context of the system you're being asked to improve and can encompass topics such as the team delivering the system and their organisational maturity (think DORA metrics), the nature of the traffic being handled, whether read or write heavy, expectations around data consistency (CAP theorem).

In a real production incident, nobody is scoring you on the elegance of your target architecture. The system is down, or degrading, and people are losing money or trust or both for every minute it stays that way. The job in that moment isn't to design the system you'd build with six months and a clean slate; it's to stop the bleeding. Roll back the last deploy. Fail over. Shed load. Kill the runaway job. Turn the feature off. Whatever gets you back to a working state fastest, even if it's ugly, even if it's temporary.

Scaling is a design activity. Restoring service is an incident response activity. They have different goals, different time horizons, and often different answers - the fastest way to get healthy again is rarely the same as the best long-term architecture.

I think this is exactly why interviewers plant a failure scenario in the question rather than just asking "design a system that can handle X requests per second." They want to see whether you notice that the priority has changed. Do you say "first, I'd want to understand what's actually broken and get it stable" before you say anything about read replicas? That one sentence tells the interviewer more about how you'd behave on a 3am page than the rest of the answer combined.

Only once the system is back on its feet does it make sense to ask why it fell over, and whether that failure mode should be designed out. That's a real and necessary conversation - just not the first one.

So when the question is framed as a failure, treat it as one. Stabilise first. Design second.

Enjoyed this post? Share it:

Share on LinkedIn