Overview

Sources: DDIA Ch. 1 · Kimball Ch. 1–4

Goal: Build a solid mental model of data engineering — how data is stored, moved, transformed, and queried at scale.

DDIA — three things that matter

Kleppmann frames every data system around three properties:

• Reliability — works correctly even when faults happen (hardware, software, human error)
• Scalability — handles growth in data volume, traffic, or complexity
• Maintainability — people can work on the system productively over time (operability, simplicity, evolvability)

There are no perfect solutions — only trade-offs tuned to your access patterns and constraints.

Two access patterns

• OLTP — many small reads/writes by key; normalized schemas; transactions matter
• OLAP — large scans and aggregates; columnar/warehouse models; throughput over latency

Kimball — analytics starts with the business process

Dimensional modeling maps business processes (sales, shipments, web clicks) to measurable facts surrounded by descriptive dimensions. The warehouse is organized for understandability and query performance, not 3NF purity.

How this wiki is organized

• Core concepts — DDIA + Kimball ideas (models, distribution, batch/stream)
• Core technologies — tools that implement those ideas (Spark, Kafka, Airflow…)