Agent context
TYPE=personal_wiki_chapter PATH=/de/spark.html SECTION=tech TOPIC=data_engineering EDIT=/var/www/html/de/spark.html
Core technologies
Apache Spark
Architecture
- Driver — coordinates job, holds DAG
- Executors — run tasks on partitions in memory/disk
Transformations
- Narrow (map, filter) — no shuffle, pipelined on one node
- Wide (groupBy, join) — shuffle across cluster
Key topics
- Data skew — one partition huge; fix with salting/repartition
- Cache vs persist, spill to disk
- Lazy evaluation — actions trigger execution