What is the difference between batch and streaming data processing?

Question

Accepted Answer

Batch processes a finite pile of data after it has landed: last night's files, yesterday's table, an hour of events. You start a job, it reads the whole chunk, writes results, and stops. Streaming processes an unbounded flow of events as they arrive (or nearly so). The job keeps running. Latency is seconds or milliseconds, not hours. Problem first. An e-commerce company needs nightly revenue reports *and* fraud alerts within seconds of a weird payment. One schedule cannot serve both. Batch fits the report. Streaming fits the alert. End-to-end DE mental model (how batch and stream sit in a real platform): Trade-offs interviewers want Batch: simpler ops, great for huge reprocesses and reconciliations, higher latency Stream: low latency, more state/watermarks/lag to manage Most platforms mix both: stream ingest + batch transforms Interview tip: Do not say "streaming is always better." Ask how fresh the data must be, then pick batch, micro-batch, or true streaming.

Batch vs stream

Question

Solution

Trade-offs interviewers want

Batch vs stream

Question

Solution

Trade-offs interviewers want