When a product manager asks for a real-time dashboard, you decide whether to build a continuous streaming pipeline by evaluating the business decision being made and translating real-time into an objective latency Service Level Agreement (SLA). In most business scenarios, decision-makers review dashboards periodically throughout the day, meaning a micro-batch pipeline running every five to fifteen minutes delivers sufficient freshness at a fraction of the cost. True continuous streaming introduces always-on infrastructure expenses, state checkpointing overhead, and 24/7 on-call operational burdens that are only justified when automated systems or frontline teams require sub-second reactions.
Investigating the decision and latency numbers
Your initial conversation with the product manager must focus on operational workflows rather than engineering jargon:
- Identify the human or automated action: Ask what immediate decision depends on the dashboard updating in seconds. If a marketing manager checks campaign revenue once every morning, updating numbers every three seconds provides zero business value. If an operations center monitors delivery fleet routes to divert drivers during road closures, sub-minute latency is genuinely required.
- Pin down numerical targets: Replace vague words like real-time with concrete time measurements. Ask: "If data arrives within ten minutes of happening, does that harm business performance?" If stakeholders confirm ten minutes is acceptable, you do not need continuous streaming.
The hidden costs of continuous streaming
Junior engineers often assume streaming is universally superior to batch, but continuous pipelines incur heavy operational overhead:
- Always-on infrastructure costs: Streaming engines like Apache Flink or Spark Structured Streaming require persistent worker nodes running 24/7. Even during low-traffic night hours, compute clusters remain active and billed. Micro-batch pipelines, by contrast, run ephemeral tasks that shut down upon completion.
- State and watermark complexity: Streaming pipelines require managing distributed state backends (such as RocksDB), configuring checkpoint storage, handling out-of-order late events, and tuning watermarks.
- On-call operational burden: A stuck streaming pipeline triggers pages at 3:00 AM because message lag accumulates rapidly in Kafka. A batch pipeline scheduled every fifteen minutes provides built-in retry windows and easier recovery.
The hybrid architecture approach
When specific metrics really require low latency while broader analytics do not, implement a hybrid design:
Raw Events (Kafka) | +---> Streaming Engine (Flink/Kafka Streams) ---> Redis/Fast Table (Alerts, Active Users) | +---> Lake Storage (Object Store) ---> 15-min Batch / dbt ---> Full Warehouse Marts
Route low-latency metrics, such as live active user counts or fraud anomaly alerts, through a targeted streaming job into a fast cache. Concurrently, land raw event partitions into object storage and use micro-batches every fifteen minutes to run dimensional modeling, complex joins, and broad executive reporting. This delivers instant operational visibility where needed without overcomplicating your analytical warehouse.