Processing
streams
Time, windows, joins, and how to process each event exactly once (effectively).
→ next · ← back · N notes · F full screen
What do you do with a stream?
Write it somewhere
Keep a database, cache, or search index up to date.
Push it to people
Alerts, notifications, live dashboards.
Derive another stream
Filter, enrich, aggregate, join: a pipeline of operators.
Reasoning about time
Event time is not processing time.
When did it happen, and when did we hear about it?
A phone queues events offline and sends them hours later. A consumer restarts and works through a backlog. Grouping by processing time gives nonsense.
Group by event time, and decide what to do with stragglers: ignore them, or publish a correction.
Three timestamps
When it happened (device clock), when it was sent (device clock), when it was received (server clock). The last two estimate the device’s clock offset.
Lab: four kinds of window
Stream joins
Joins that never finish.
Three kinds of stream join.
Stream–stream
Match a search with clicks on its results within one hour. Keep recent events of both sides as state.
Stream–table
Enrich each payment with merchant details. Keep a local copy of the table, updated through change data capture.
Table–table
Maintain a materialised view, like the Twitter home timeline cache from session 1.
The catch: if the joined data changes over time, which version applies? The fee rate at the time of the payment, not today’s.
Fault tolerance
Exactly once, effectively.
Crash between “done” and “recorded as done”.
Now, our system.
Late transactions
What happens to a transaction that arrives after its settlement window closed? Ignored, moved to the next window, or corrected?
Fees by time
Do we always apply the fee plan that was valid at the payment’s time, even if it changed since?
Consumers and duplicates
Which of our consumers would double-count if a message were redelivered?
Event time vs server time
Do we use the PSP’s timestamp or our receive time for cut-offs? Do we store both?
The future of data systems
Integrating everything, databases turned inside out, and building for correctness. Chapter 12.