Agent context
TYPE=personal_wiki_chapter PATH=/de/modeling.html SECTION=concepts TOPIC=data_engineering EDIT=/var/www/html/de/modeling.html
Core concepts
Data modeling
Sources: Kimball Ch. 2–7 · DDIA Ch. 2–3
Always ask: Who queries this, how often, with what tools? Kimball: model the business process, not the source system layout.
Kimball — dimensional modeling basics
Fact tables
- One row = one measurement at a declared grain (e.g. one line item on an order)
- Contain foreign keys to dimensions + numeric facts (amounts, counts)
- Grain must be atomic and consistent — never mix grains in one fact table
Dimension tables
- Descriptive context: who, what, where, when, why
- Wide, denormalized, human-readable (often include labels, hierarchies)
- Examples: date, customer, product, store
Star schema
- One fact surrounded by dimensions — looks like a star
- Snowflake = normalized dimensions (more joins, less redundancy)
- Star is preferred for simplicity and BI tool performance
Grain — the most important decision
Grain answers: "Exactly what does one row represent?" Get this wrong and every metric is wrong. Examples:
- Order line item · Daily store sales · Individual web page view
Conformed dimensions & bus architecture (Kimball)
- Conformed dimension — same dimension table (same keys, attributes) reused across multiple fact tables
- Enables drill-across — compare metrics from different processes on shared dimensions (e.g. same Date dim for sales + inventory facts)
- Enterprise bus matrix — grid of business processes × conformed dimensions; integration blueprint
Slowly changing dimensions (SCD)
- Type 1 — overwrite (no history)
- Type 2 — new row with effective dates + current flag (full history)
- Type 3 — limited history column (previous value)
DDIA perspective — schema on read vs write
- Schema-on-write (relational warehouse) — enforce structure at load; safe, governed analytics
- Schema-on-read (data lake) — structure applied at query time; flexible, messier
Star schema vs OBT (modern warehouses)
- Star (Kimball) — normalized dims, many joins; best for consistency and BI semantics
- One Big Table — pre-joined wide table; eliminates join cost on MPP engines; duplication acceptable when storage is cheap