Pipelines & scenarios · Core Pipeline Concepts

What is streaming processing?

Easystreamingreal-timeKafkalatencymicro-batch

Interview question

What is streaming processing in data engineering?

How to answer

The short answer

Streaming processing means handling events continuously (or in tiny micro-batches) as they arrive, instead of waiting for a daily file.

Typical use cases

  • Fraud / anomaly alerts within seconds
  • into a lake or search index
  • Real-time feature updates for ML
  • Live ops dashboards (lag of seconds/minutes)

What gets harder

State, late data, watermarks, exactly-once semantics, consumer lag, and ops complexity.

Hybrid reality

Many platforms stream into Kafka or a lake, then run batch dbt models on a schedule. Streaming and batch are tools, not religions.

Interview tip: Pick streaming when latency is a product requirement. Otherwise prefer batch for simplicity. Say "how fresh must this be?" before naming Kafka.