MPP & cloud warehouses

Sources: DDIA Ch. 5–6 (replication, partitioning) · Ch. 3 (storage engines)

Why distribution exists (DDIA)

Single-node limits: disk, memory, CPU. Scale out by splitting data across machines. Two fundamental strategies:

• Replication — copy data to multiple nodes (read scaling, fault tolerance)
• Partitioning (sharding) — split data by key across nodes (write/read scaling)

Replication (DDIA Ch. 5)

• Leader/follower — writes go to leader, followers replicate; common in Postgres, MySQL
• Multi-leader — writes on multiple nodes; conflict resolution needed
• Leaderless — quorum reads/writes (Dynamo-style)
• Trade-off: consistency vs availability vs latency — no free lunch

Partitioning (DDIA Ch. 6)

• Each partition owns a subset of keys; routing via partition key hash or range
• Skew — hot keys overload one partition (celebrity user problem)
• Rebalancing — move partitions when cluster grows; avoid too many small partitions
• Joins across partitions require expensive shuffles over the network

OLTP vs OLAP

• OLTP — row-oriented, indexed point lookups, short transactions
• OLAP / warehouse — column-oriented, scan-heavy aggregates; MPP query engines
• Modern cloud warehouses decouple storage from compute (Snowflake, BigQuery)

Column storage (DDIA Ch. 3)

Columnar formats (Parquet, ORC) compress well and read only needed columns — ideal for analytics scans. Row stores win for point lookups and writes.