Designing Data-Intensive Applications, session 5

Encoding and
evolution

Old and new code will run at the same time, reading each other’s data. Plan for it.

→ next · ← back · N notes · F full screen

Old and new versions always overlap.

Servers roll out node by node. Mobile users don’t update for months. Stored data outlives the code that wrote it.

Backward compatible

New code can read data written by old code. Usually easy: you know the old format.

Forward compatible

Old code can read data written by new code. Harder: it must ignore what it doesn’t understand.
01

Formats

From Java serialisation to Avro.

Formats to avoid, and formats to use carefully.

Language built-ins

Java Serializable, Python pickle: locked to one language, slow, poor versioning, and decoding can instantiate arbitrary classes, a classic security hole.

JSON, XML, CSV

Human-readable and everywhere. But: numbers are ambiguous (IDs above 253 break in JavaScript), no binary strings (Base64 adds 33%), schemas are optional.

Binary JSON variants

MessagePack, BSON: a bit smaller, but field names still repeat in every record.

Same record, how many bytes?

JSON (minified)81 B
MessagePack66 B
Thrift Binary59 B
Thrift Compact34 B
Protocol Buffers33 B
Avro32 B

The binary schema-based formats drop field names entirely. They store a tag number (Thrift, Protobuf) or nothing at all (Avro).

Two schema philosophies.

Protocol Buffers: tags identify fields
message Settlement {
  required int64  id          = 1;
  optional string merchant    = 2;
  repeated string fee_codes   = 3;
}
Avro IDL: no tags at all
record Settlement {
  long          id;
  union { null, string } merchant = null;
  array<string> feeCodes;
}

Protobuf / Thrift

Each field has a tag number in the bytes. Code generation from the schema.

Avro

Bytes are just values in schema order. Decoding needs the writer’s schema, resolved against the reader’s schema.

02

Schema evolution

Which changes are safe?

Lab: will this change break anything?

Backward compatible: new code reads old data
Forward compatible: old code reads new data

Schemas are worth it.

Smaller
no field names in every record
Documented
the schema is always up to date, unlike docs
Checked
compatibility can be tested before you deploy

Plus code generation for statically typed languages like Java.

03

Modes of dataflow

Through databases, services, and messages.

Through a database: data outlives code.

Through services: RPC isn’t a local call.

Local function call
Network request
Succeeds or throws
May time out with no idea whether it ran
Retrying is safe
Retry may run it twice unless it’s idempotent
Takes about the same time each call
Latency varies wildly
Pass pointers to objects
Everything must be encoded into bytes
Same language both sides
Types must translate between languages

Through messages: send it and move on.

Buffer

The broker absorbs bursts and outages of the consumer.

Redeliver

Messages to a crashed process can be delivered again.

Decouple

The sender doesn’t need the receiver’s address, and one message can reach many consumers.

Actor frameworks (Akka, Orleans, Erlang) pass messages too, so the same forward/backward compatibility rules apply during rolling upgrades.

Now, our system.

Java serialisation

Is Serializable or a similar format used anywhere: caches, sessions, queues? It’s both fragile and a security risk.

Rolling deploys

When we add a column or JSON field, can the previous version still run safely next to the new one?

Whole-entity saves

Does JPA write back full entities? Could an old instance overwrite columns it doesn’t know?

API versions

How do we evolve the settlement API and webhook payloads without breaking partners?

Next session

Replication, part 1

Leaders, followers, and the strange things replication lag does to your users. Chapter 5.