Kafka · Advanced Kafka

What is Kafka Streams?

Mediumkafka streamsstream processingstate storedsl

Interview question

What is Kafka Streams, and how does it differ from a plain consumer?

How to answer

The short answer

Kafka Streams is a Java/Scala client library for building stream processing applications on Kafka. Your app is both a consumer and a producer, with higher-level DSL/Processor APIs.

What you get beyond a raw consumer

  • Declarative operations: filter, map, join, aggregate, windowing
  • State stores (local RocksDB) for aggregations and joins
  • Fault-tolerant state via changelog topics
  • Scaling by running more stream thread / app instances ( assignment under the hood)
  • Optional exactly-once processing guarantee

Mental model

You write an application that embeds the Streams library. There is no separate Streams cluster. Scale-out = run more instances of your app with the same application.id.

vs plain consumer

Plain consumerKafka Streams
You hand-code poll/process/produceDSL for transforms
You manage state yourselfManaged state stores
You wire joins manuallyStream-table joins built-in

Streams is library-embedded, lightweight for many Kafka-centric services. Spark/Flink are fuller processing platforms with different ops models.

Interview tip: "Kafka Streams = embeddable stream processing library on Kafka, with state and optional EOS; scale by adding app instances."