Many leaders,
or none at all
When more than one node accepts writes, conflicts become your problem.
→ next · ← back · N notes · F full screen
Multi-leader
Every datacenter, device, or editor is a leader.
When does it make sense?
Several datacenters
A leader per region: local write latency, and a region can fail or lose its link without stopping writes elsewhere.
Offline clients
A calendar app on your phone accepts edits offline. Every device is effectively its own datacenter.
Collaborative editing
Google Docs style: each user’s local copy accepts edits and syncs later.
Often bolted on after the fact, and it interacts badly with auto-increment keys, triggers, and constraints. Avoid it if you can.
Write conflicts arrive late.
Resolving conflicts.
Avoid them
Route all writes for one record to the same leader, e.g. the user’s home region. The most common recommendation.
Converge
Last write wins by timestamp or ID (simple, loses data), or merge values, or keep both and let code or a human decide.
Merge automatically
CRDTs, mergeable data structures, operational transformation (used by collaborative editors).
Leaderless
Write to several replicas. Read from several. Vote.
Dynamo-style replication.
The client (or a coordinator) sends every write to all n replicas and waits for w acks. Reads ask r replicas and take the newest version.
Read repair
A read that sees a stale replica writes the new value back to it.
Anti-entropy
A background process compares replicas and copies missing data.
Lab: quorums
Even with w + r > n, edge cases leak.
- A sloppy quorum writes to stand-in nodes during an outage (hinted handoff), so the overlap guarantee is gone.
- Two concurrent writes: which one wins isn’t clear, and last-write-wins by clock can drop one.
- A write concurrent with a read may or may not be seen.
- A write that reached fewer than w nodes is reported as failed but is not rolled back.
- A node holding the new value fails and is restored from an older one.
Quorums give probabilities, not guarantees. Monitoring staleness is also harder than measuring a follower’s lag.
Concurrent writes
“Concurrent” has nothing to do with clocks.
Capturing happens-before with versions.
Now, our system.
Multi-region?
Do we write to more than one region today, or plan to? Which records would conflict?
Offline clients
Do any apps (mobile, back-office tools) let users edit offline and sync later?
LWW hiding in plain sight
Where do we overwrite by “latest updated_at wins”, e.g. syncing partner data or caches?
Merging, not overwriting
Which of our updates could be modelled as merges or append-only records instead of overwrites?
Partitioning
Splitting data across machines: hot spots, secondary indexes, and rebalancing. Chapter 6.