The future of
data systems
Putting it all together: integrating tools, unbundling databases, and making systems correct.
→ next · ← back · N notes · F full screen
Data integration
No single tool does everything.
Derived data instead of distributed transactions.
Write once to a system of record. Everything else is derived from its log: indexes, caches, warehouses.
Need to reprocess everything for a new schema? Run the old and new derived views side by side and migrate gradually.
Unbundling databases
Your whole organisation is one big database.
A database, turned inside out.
Indexes are derived views
Inside a database, CREATE INDEX derives a structure from the table and keeps it in sync. Across systems, change streams do the same job.
Application code as a function
Stream processors transform one dataset into another, like spreadsheet formulas that update when inputs change.
Subscribe, don’t query
Instead of asking the currency service for a rate on every payment, subscribe to rate changes and keep a local copy.
Every cache and materialised view moves the line between work on write and work on read. Remember Twitter.
Aiming for correctness
End to end, or not at all.
Lab: the end-to-end argument
Integrity matters more than timeliness.
Timeliness
Users see the latest state. Violations are temporary: wait and try again.
Integrity
No corruption, nothing lost, nothing double counted. Violations are permanent.
Apologise, then fix
Many constraints can be checked after the fact with compensating actions, like airlines overbooking or banks charging overdraft fees.
Uniqueness in a log-based system: route all requests for one key to one partition, and a single consumer decides in log order.
Trust, but verify.
Hardware and software are not perfect. Data gets corrupted silently. Mature systems audit: check the integrity of data continuously, end to end.
Event-based designs make this easier: rerun the derivation and compare.
Doing the right thing
Data about people is powerful. Predictive systems can encode bias and create feedback loops. Collect only what you need, ask for real consent, and treat data as a liability as well as an asset.
Now, our system.
Idempotency end to end
Do all money-moving APIs accept an idempotency key from the client, and is it checked with a unique constraint?
Systems of record
For each piece of data, which system is the source of truth, and which are derived copies?
Reconciliation as auditing
Do we continuously reconcile ledger totals, bank statements, and payouts, or only when someone complains?
Data we shouldn’t keep
What personal or card-related data do we store that we don’t actually need?
That’s the whole book.
Seventeen sessions, one idea underneath: know what your data system guarantees, and what it doesn’t.