Replication:
leaders and followers
Keeping copies of the same data on several machines is easy until the data changes.
→ next · ← back · N notes · F full screen
Why keep copies?
All the difficulty is in handling changes. Three families: single-leader, multi-leader, leaderless.
Leaders and followers
One node takes writes. Everyone else copies.
Synchronous or asynchronous?
When nodes go down.
Add a follower
Take a consistent snapshot, copy it, then replay the log from the snapshot’s exact position (PostgreSQL: log sequence number).
Follower fails
On restart it knows the last transaction it processed and catches up from the leader’s log.
Leader fails: failover
Detect it (timeout), pick the most up-to-date follower, reconfigure clients and the old leader.
Failover is where things break: lost async writes, split brain with two leaders, and timeouts that are either too slow or trigger on a busy-but-alive leader.
What exactly gets sent to followers?
Statements
Replay each SQL statement. Breaks on NOW(), RANDOM(), auto-increments, and side effects.
Write-ahead log
Ship the storage engine’s own log (PostgreSQL, Oracle). Precise, but tied to the on-disk format, so versions must match.
Logical (row) log
Describe changed rows independently of storage (MySQL binlog). Lets leader and followers run different versions, and feeds change data capture.
Replication lag
Eventually consistent, for some value of eventually.
Lab: reading from followers
Three guarantees users quietly expect.
Read-after-write
You always see your own updates. Read things the user can edit from the leader, or remember the time of their last write and avoid stale replicas.
Monotonic reads
You never see time go backwards. Route each user to the same replica, for example by hashing the user ID.
Consistent prefix
You see writes in the order they happened: never an answer before its question. Keep causally related writes in the same partition.
Pretending async replication is synchronous is a recipe for bugs. This is what transactions are for (sessions 9–13).
Now, our system.
Who reads from replicas?
Which screens or jobs read from a PostgreSQL replica? Could any of them show an operator stale data right after they changed it?
Failover
How do we fail over today: automatic or manual? What happens to writes the old primary accepted but never shipped?
Sync settings
Is any replica synchronous? Do we know our typical and worst replication lag?
External side effects
If a failover discards recent writes, have we already sent emails, webhooks, or bank files based on them?
Replication, part 2
Multiple leaders, no leader at all, quorums, and concurrent writes. Chapter 5.