Data orchestration
Running data jobs in the right order, at the right time, and handling it when they fail.
A real data platform is dozens or hundreds of jobs: pull orders from the shop database, pull payments from the bank, clean both, join them, rebuild the data mart, refresh the dashboards. Each step depends on others finishing first. Data orchestration is the layer that knows those dependencies, starts each job when its inputs are ready, and keeps track of what ran, what failed and what is late.
Most orchestrators describe a pipeline as a DAG, a directed acyclic graph: boxes for tasks and arrows for "this must finish before that starts", with no loops. You write the DAG as code, the orchestrator schedules it (every night at 2 a.m., or whenever a new file lands), runs independent tasks in parallel, retries the ones that fail, and alerts someone when a retry is not enough. If the payments extract breaks, nothing downstream runs on half the data.
Apache Airflow is the best-known tool. It started at Airbnb in 2014, became an Apache project, and pipelines in it are written in Python. Newer tools like Dagster and Prefect put more weight on the data each task produces rather than only the tasks, and clouds offer managed versions such as Amazon MWAA. Orchestration does not transform the data itself; it calls the tools that do, such as dbt, Spark or plain SQL, and makes sure they run in order.
Keep in mind
- Runs data jobs in dependency order, on a schedule or on events.
- Pipelines are usually DAGs: tasks plus "runs after" arrows, no loops.
- Handles retries, alerts, backfills and a history of every run.
- Apache Airflow is the classic tool; Dagster and Prefect are newer options.
