Designing Data-Intensive Applications, session 14

Batch processing

Big, bounded input in. Derived output out. Repeat whenever you like.

→ next · ← back · N notes · F full screen

Three kinds of system.

Services
online: request in, response out; judged by latency
Batch jobs
offline: big input, run periodically; judged by throughput
Streams
near real time: between the two
01

The Unix philosophy

Small tools, one interface, composed.

Top 5 most-called endpoints, in one line.

cat access.log |
  awk '{print $7}' |      # take the URL field
  sort             |      # bring identical URLs together
  uniq -c          |      # count each run
  sort -rn         |      # most frequent first
  head -n 5

Do one thing well

Each program is small, and they combine in ways no one planned.

Uniform interface

Everything reads and writes lines of text on stdin and stdout.

Easy to experiment

Inputs are never modified, so you can rerun and inspect each stage.

02

MapReduce

Unix pipes across a thousand machines.

Lab: one MapReduce job

Count payments per status. Each line is one input record.
1. Map: emit (key, 1)
2. Shuffle: sort and group by key
3. Reduce: sum per key

Input lives in a distributed filesystem (HDFS). Mappers run where the data is. The framework sorts and moves data between them: the shuffle.

Joins without random lookups.

payments (events) merchants (DB dump) map: key = merchant map: key = merchant reducer for “acme”:merchant row, then its events

Sort-merge join: key both inputs by the join key. The shuffle delivers each merchant’s record and all its events to the same reducer.

No network round trip per record. Hot keys? Spread them over several reducers.

Batch outputs, and why they’re safe.

Build indexes and stores

Search indexes, or whole key-value database files built in bulk and swapped in atomically.

Immutable inputs

A buggy job can be fixed and rerun. Old output can be kept for rollback.

Retry is safe

Failed tasks rerun on another machine; partial output is discarded.

Human fault tolerance: bad code doesn’t destroy data, it just produces output you can throw away.

03

Beyond MapReduce

Fewer disk writes, smarter plans.

Dataflow engines: Spark, Flink, Tez.

MapReduce
Dataflow engines
Every job writes full output to HDFS
Intermediate state stays in memory or local disk
Sort between every stage
Sort only when needed
Each job starts from scratch
The whole workflow is one optimised graph
Fault tolerance via materialised files
Recompute lost partitions from inputs (needs determinism)

On top: declarative APIs (Spark DataFrames, Hive, Pig) with query optimisers. Graph jobs use Pregel-style message passing.

Now, our system.

Our nightly jobs

Can the settlement batch be rerun for a given day and produce the same result? What if it ran twice?

Immutable inputs

Do we keep the exact input of each run (transactions, bank files) so we can reproduce it?

Reconciliation is a join

Matching our records with the bank’s file is a join. Is it done row by row with queries, or as a sorted merge?

Skew

Do one or two large merchants make a single batch step much slower than the rest?

Next session

Streams, part 1

Message brokers, logs, change data capture, and event sourcing. Chapter 11.