Batch vs stream processing explained for beginners, with a SQL example

Think about two ways to count the cash in a shop. At closing time the owner empties the till and counts everything once: that is batch. Or a screen above the counter adds each sale to a running total the moment it is rung up: that is streaming. Same sales, same total at the end of the day. The difference is how fresh the number is while the day is still happening, and how much effort it takes to produce it.

Batch processing

In batch processing, data collects over a period and is processed together on a schedule: every night, every hour, every 15 minutes. A batch job reads a whole chunk of data, processes it, writes the result, and stops.

Batch is the default for most reporting. It is simpler to build, easy to rerun if something goes wrong (run it again on the same chunk), and cheaper, because the computer works only while the job runs. Its weakness is delay: the result is only as fresh as the last run.

Stream processing

In stream processing, each event, such as a click, a payment or a sensor reading, is handled soon after it happens, one at a time or in tiny groups. The program never "finishes", because new events keep arriving. The result is always close to up to date, usually within seconds.

Streaming is needed when delay costs something: a fraud alert must reach you before the payment completes, a delivery app must show where the rider is now, a monitor must warn when a server fails. The price is complexity: you must handle events that arrive late or twice, and keep the program running all the time.

The same 6 events, both ways

Six payments came in during 1 day. The queries ran in DuckDB, and each output is exactly what it printed.

CREATE TABLE payments (event_id INTEGER, event_time TIMESTAMP, amount INTEGER);

INSERT INTO payments VALUES
  (1, '2026-03-01 09:05:00', 1200),
  (2, '2026-03-01 09:40:00',  800),
  (3, '2026-03-01 11:15:00',  450),
  (4, '2026-03-01 14:30:00', 2500),
  (5, '2026-03-01 14:50:00',  300),
  (6, '2026-03-01 18:20:00',  950);

The batch view. At the end of the day, one job reads everything and produces one answer:

SELECT COUNT(*) AS payments, SUM(amount) AS day_total
FROM payments;

payments | day_total
---------+----------
6        | 6200
(1 row)

1 row, computed once, available after midnight. Before the job ran, nobody knew the total.

The streaming view. The total is kept up to date as each event arrives, so after every payment there is a new number. A running total shows how that looks:

SELECT event_id, event_time, amount,
       SUM(amount) OVER (ORDER BY event_time) AS total_so_far
FROM payments
ORDER BY event_time;

event_id | event_time          | amount | total_so_far
---------+---------------------+--------+-------------
1        | 2026-03-01 09:05:00 | 1200   | 1200
2        | 2026-03-01 09:40:00 | 800    | 2000
3        | 2026-03-01 11:15:00 | 450    | 2450
4        | 2026-03-01 14:30:00 | 2500   | 4950
5        | 2026-03-01 14:50:00 | 300    | 5250
6        | 2026-03-01 18:20:00 | 950    | 6200
(6 rows)

After event 1 the total is 1200, after event 2 it is 2000, and so on, until 6200 at the last one. The final number is the same as the batch result, but you could have read it at any moment of the day. A real streaming system updates the number as each event comes in, instead of recalculating it from the start.

Micro-batch sits in between. Group events into small time windows, such as 1 hour, and process each window as soon as it closes:

SELECT date_trunc('hour', event_time) AS hour,
       COUNT(*)                       AS payments,
       SUM(amount)                    AS revenue
FROM payments
GROUP BY date_trunc('hour', event_time)
ORDER BY hour;

hour                | payments | revenue
--------------------+----------+--------
2026-03-01 09:00:00 | 2        | 2000
2026-03-01 11:00:00 | 1        | 450
2026-03-01 14:00:00 | 2        | 2800
2026-03-01 18:00:00 | 1        | 950
(4 rows)

Four windows with results: 9 am has two payments, 11 am one, 2 pm two, and 6 pm one. Each window's answer is ready soon after the hour ends. Many systems that call themselves "streaming" actually work like this.

The hard parts of streaming

Beginners do not need to solve these yet, but knowing they exist explains why streaming is harder.

• Late events. A payment made at 9:50 may reach the system at 10:10. A batch job simply includes it the next time it runs. A streaming job must decide how long to wait before closing a window.
• Duplicates. Networks resend messages, so the same event can arrive twice and must not be counted twice.
• Always on. A batch job that fails can be rerun in the morning. A streaming job that fails means events pile up or are lost, so it needs monitoring around the clock.

Batch vs streaming at a glance

Which one should a beginner learn first?

Batch. Nearly every data engineering job uses it, it is easier to reason about, and the ideas you learn (cleaning, joining, checking totals, safe reruns) carry over to streaming. Learn it from what is a data pipeline, and then move to streaming with what is Apache Kafka. When someone asks you "do we need real time?", the useful reply is another question: "what decision changes if the number is 1 hour old?" If the answer is "none", batch is enough.