Batch vs streaming

Sources: DDIA Ch. 10–11 · Kimball Ch. 16 (ETL)

Two ways data flows (DDIA)

• Batch — process bounded, immutable datasets (nightly jobs, historical backfills)
• Stream — process unbounded event streams continuously (real-time dashboards, alerts)

Batch processing (DDIA Ch. 10)

• Input is large and fixed; output is derived dataset (MapReduce, Spark batch, SQL INSERT…SELECT)
• Unix pipeline philosophy: immutable files, recomputation, simple failure recovery
• High throughput; latency measured in minutes to hours

Stream processing (DDIA Ch. 11)

• Events arrive continuously; state updated incrementally
• Event time vs processing time — watermarks handle late-arriving data
• Window types: tumbling, hopping, session
• Harder correctness: idempotency, dedup, exactly-once semantics

The log as central abstraction (DDIA)

An append-only log (Kafka, WAL) decouples producers and consumers. Same log can feed batch reprocessing and stream consumers — foundation of unifying batch + stream.

Lambda vs Kappa

• Lambda — separate batch (accuracy) + speed (latency) layers; two codepaths to maintain
• Kappa — stream-only; reprocess history by replaying the log

CDC & keeping systems in sync (DDIA Ch. 11)

Change Data Capture turns DB writes into a stream — warehouse stays in sync with operational systems without full reloads.

Kimball ETL perspective

ETL subsystems: extract → clean/conform → deliver. Conform dimensions to the bus architecture before loading facts. Batch remains the default for warehouse loads; streaming adds a speed layer when latency matters.