Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · UDFs & Streaming

What is Structured Streaming?

Hardstructured-streamingstreamingkafka

Interview question

What is Structured Streaming in Spark?

How to answer

The short answer

Structured Streaming is Spark's high-level streaming engine that treats a live data stream as an unbounded table. You write almost the same /SQL transformations as batch; Spark runs them incrementally.

Model

Output
Source (Kafka/files/socket)
   -> continuous Input Table
   -> query (filter/join/agg)
   -> Result Table
   -> Sink (Kafka/Parquet/foreachBatch)
        with trigger (micro-batch or availableNow)

Example

Python
stream = (
    spark.readStream
    .format("kafka")
    .option("subscribe", "orders")
    .load()
)

parsed = parse_orders(stream)  # your parsing

query = (
    parsed.writeStream
    .format("parquet")
    .option("path", "/lake/bronze/orders")
    .option("checkpointLocation", "/chk/orders")
    .outputMode("append")
    .start()
)
query.awaitTermination()

Output modes

  • append: new rows only (common for raw ingest)
  • complete: full result rewritten (some aggs)
  • update: changed rows (supported sinks)

Interview tip

Emphasize checkpointing for fault tolerance, exactly-once sinks when supported, and that micro-batch is the common execution mode.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill