Batch processing
Collecting data for a while and then processing it all in one big run.
Batch processing means handling data in large chunks on a schedule, instead of one item at a time as it arrives. A bank collects the day's transactions and settles them overnight. A company computes payroll once a month. A data team loads yesterday's sales into the data warehouse every morning at 2 AM so the dashboards are fresh by 9.
The idea is older than screens: early computers ran stacks of punched cards as one batch. It survives because it is simple and efficient. A single big run can use machines fully and then shut them down, it is easy to rerun if something fails, and the results are consistent because everything was computed from the same snapshot. Most ETL pipelines and tools like Apache Spark are typically run this way, often triggered by a cron job or an orchestrator.
The cost is delay: a batch report is only as fresh as the last run. When minutes or seconds matter, such as fraud detection or live tracking, teams switch to stream processing, which handles each event as it comes. Many companies use both: streams for what must be immediate, batches for heavy, complete calculations.
Keep in mind
- Processes accumulated data in scheduled runs, often nightly.
- Efficient and easy to rerun, but results are only as fresh as the last run.
- The opposite approach is streaming, which handles each event as it arrives.
- Typical uses: payroll, billing, nightly ETL loads, monthly reports.
