Event streams and
change data capture
Batch processing with the input never ending. And databases turned inside out.
→ next · ← back · N notes · F full screen
An event: something happened, at some time.
A producer writes it once; many consumers may process it. Related events are grouped into a topic. Instead of consumers polling a database, the system notifies them.
Transmitting events
Message brokers, two styles.
Two questions decide everything.
Producers faster than consumers?
Drop messages, buffer them in a queue, or apply backpressure and block the producer.
A node crashes?
Are messages lost, or written to disk and replicated? Durability costs throughput.
Direct or via a broker?
UDP, ZeroMQ, webhooks are direct and lossy. A broker (RabbitMQ, ActiveMQ, Kafka) is a database tuned for messages.
Log-based brokers: messages as an append-only log.
Databases and streams
Your database already has a log.
Dual writes race each other.
Fix: one system is the leader, and the others follow its change log.
Change data capture.
Capture every change from the database’s replication log and publish it as a stream. Search indexes, caches, and warehouses become followers of the database.
Initial snapshot
Start with a consistent snapshot tied to a log position, then follow the log.
Log compaction
Keep only the latest value per key, so a new consumer can rebuild from the log alone.
Event sourcing: store what happened, not just where you ended up.
State and immutability
The log is the truth. Tables are a cache.
Accountants figured this out centuries ago.
A ledger is append-only. Mistakes aren’t erased; they’re corrected with a new entry. The balance is derived from the entries.
State is the integral of the event stream over time.
Benefits
Full audit trail, recover from buggy code by replaying, many read-optimised views from one log (CQRS).
Limits
Heavy churn makes logs huge; legal deletion requests need real deletion (excision), not just a new event.
Now, our system.
Dual writes
Where do we write to PostgreSQL and then to Kafka, Redis, a search index, or a partner API in the same code path?
The ledger
Is our ledger truly append-only, with corrections as new entries? Or do we update balances in place?
CDC from PostgreSQL
Could logical replication or an outbox table replace hand-written publishing code?
Replay
If a bug miscalculated fees for a week, could we replay events to rebuild correct balances?
Streams, part 2
Windows, stream joins, and exactly-once processing. Chapter 11.