Data engineering
The work of building the pipes that move data to where people can use it.
Data engineering is the discipline of collecting data from the places it is born, cleaning and reshaping it, and delivering it somewhere reliable where others can use it. A company's data starts scattered: the app's database, the payments provider, the CRM, spreadsheets, event logs. A data engineer builds and runs the pipelines that bring all of that together, usually into a data warehouse or a data lake.
Day to day, that means writing ETL or ELT jobs in SQL and Python, scheduling them with tools like Airflow, modeling tables with dbt, and handling large volumes with engines like Apache Spark. It also means the unglamorous part: making jobs rerun safely when they fail, noticing when a source changes its format, keeping costs under control and making sure yesterday's numbers arrive before the 9 a.m. meeting.
The role sits upstream of the analysts and data scientists. They ask the questions; data engineers make sure the answers rest on data that is complete, on time and consistent. As companies feed more of their data into AI systems, this plumbing has become more visible, because a model is only as good as the data that reaches it.
Keep in mind
- Moves data from many sources into a warehouse or lake, cleaned and consistent.
- Core tools: SQL, Python, orchestrators like Airflow, dbt, Spark.
- Much of the job is reliability: failures, late data, changed sources.
- Works upstream of analysts, data scientists and machine learning teams.
