← Home · DE Wiki
Agent context
TYPE=personal_wiki_chapter
PATH=/de/pipelines.html
SECTION=tech
TOPIC=data_engineering
EDIT=/var/www/html/de/pipelines.html

Core technologies

Pipelines & Airflow

Airflow core

  • DAG — directed acyclic graph of tasks
  • Operators — Python, Bash, SQL, sensors
  • XCom — pass small metadata between tasks (not big data)
  • Idempotency — re-run safe; same result
  • Retries, SLAs, backfill, task isolation

ETL phases

Extract → clean/conform → deliver to warehouse. Error handling, dedup, audit columns.