Reliable.Scalable.Maintainable.
Three questions to answer before you pick any database, cache, or queue.
→ next · ← back · N notes · F full screen
The bottleneck isn’t the CPU anymore.
It’s the amount of data, its complexity, and how fast it changes.
Glue code makes you a data system designer.
Put a cache and a search index next to your database, and keeping them in sync is now your job.
Three words we use daily and rarely define.
Reliability
Keeps working correctly, even when things go wrong.
Scalability
Has reasonable ways to cope as load grows.
Maintainability
Many people, over years, can work on it productively.
Reliability
Continuing to work correctly, even when things go wrong.
A fault is not a failure.
Fault: one component breaks its spec.
Failure: users stop getting service.
Faults will happen. The job is to stop them becoming failures.
Where faults come from
Break things on purpose.
Many critical bugs hide in error handling. Netflix’s Chaos Monkey randomly kills processes so the recovery path is always tested.
Designing for unreliable humans
Lab: at scale, failure is routine
Scalability
Not a label. A question about growth.
“If load grows like this, what are our options?”
First describe today’s load with a few load parameters:
Twitter, 2012: the problem was fan-out.
Work at read time, or at write time?
1. Join when reading
One global tweets table. Each timeline read runs:
SELECT tweets.*, users.* FROM tweets
JOIN users ON tweets.sender_id = users.id
JOIN follows ON follows.followee_id = users.id
WHERE follows.follower_id = :meCouldn’t keep up with 300k reads/sec.
2. Fan out when writing
A cached “mailbox” per user. Each tweet is pushed into every follower’s timeline.
Reads become cheap, and reads outnumber posts ~100×.
But 4.6k tweets × 75 followers = 345k writes/sec, and one celebrity tweet = 30M writes.
Final answer: a hybrid. Fan out for most users; merge celebrities in at read time.
Lab: fan-out calculator
Response time is a distribution, not a number.
Lab: why the mean misleads
The tail is your best customers.
Measure on the client side: a few slow requests block the fast ones behind them.
Lab: one slow call ruins the request
Never average percentiles.
To combine servers or time windows, add the histograms, then read the percentile.
No magic scaling sauce.
Real systems mix both. Stateless services spread easily; a stateful database doesn’t.
Same throughput. Completely different systems.
Maintainability
Most of the cost of software comes after launch.
Operability
Make it easy for ops to keep it healthy.
Monitoring, automation, no single-machine dependency, predictable behaviour, good defaults you can override.
Simplicity
Make it easy for a new engineer to understand.
Remove accidental complexity: the kind that comes from the implementation, not the problem.
Evolvability
Make change easy when requirements shift.
Agility, but at the level of the whole data system, not just the code.
Best weapon against complexity: good abstractions. SQL hides disk layouts, concurrency, and crash recovery.
Now, our system.
Reliability
A settlement run crashes halfway: fault or failure? Can we recompute safely? Do we detect amount mismatches automatically?
Scalability
What’s our key load parameter? Transactions per settlement window, read/write ratio, or skew across merchants (our “celebrities”)?
Performance
Do we track response times and batch durations as percentiles or averages? How much of our p99 is GC?
Maintainability
Where is our accidental complexity? How long would a new engineer need to trace one settlement into PostgreSQL?
Data models & query languages
Relational or document? The object-relational mismatch we meet daily with JPA, declarative queries, and graphs.