Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Real-time OLAP stores

Data platform · Distributed Systems Basics

Real-time OLAP stores

Mediumdata-platform-33
real-time-olapclickhouseapache-pinotapache-druid

Question

What are Druid, Pinot and ClickHouse used for?

Solution

Apache Druid, Apache Pinot, and ClickHouse are real-time analytical (OLAP) databases engineered to deliver sub-second query responses over billions of rows at concurrency rates of thousands of queries per second. They ingest streaming data directly from Apache Kafka or event brokers, applying columnar compression, inverted indexing, and pre-aggregation rollups during ingestion. You use these stores when internal dashboards or customer-facing analytical applications require low latency and high concurrency that would make cloud warehouses prohibitively slow and expensive.

High concurrency analytical serving

Cloud warehouses like Snowflake, BigQuery, and Databricks are optimized for heavy transformations, complex multi-way joins, and moderate concurrency. When exposed to external customer portals where ten thousand concurrent users refresh dashboards simultaneously, cloud warehouses trigger cluster auto-scaling, causing massive cost spikes and queuing delays.

Real-time OLAP engines solve this through specialized storage formats:

  • Apache Pinot: Built for user-facing applications (such as LinkedIn profile view analytics). It provides star-tree pre-aggregated indexes and deep integration with Kafka, delivering p95 latencies below 50 milliseconds across tens of thousands of concurrent users.
  • Apache Druid: Tailored for real-time exploratory slicing and dicing, relying on bitmap indexing, dictionary encoding, and timestamp partitioning to accelerate multi-dimensional dashboard filtering.
  • ClickHouse: A high-performance columnar engine capable of vectorized query execution and streaming ingestion, widely adopted for observability logs, financial analytics, and ad-tech attribution.
Ingestion Stream (Kafka) -> Real-time OLAP (Pinot/ClickHouse) -> 10,000+ App Users (<50ms)
Daily Lakehouse (Batch)  -> Cloud Warehouse (Snowflake)      -> Internal Analysts (3-15s)

These engines trade SQL flexibility to achieve their remarkable response speeds:

Trade-offs against warehouse batch processing

To maintain sub-second latency across massive scale, real-time OLAP engines sacrifice relational capabilities:

  • Rigid join support: Distributed shuffles required for multi-table joins are either restricted or perform poorly. Data must typically be flattened and denormalized into wide tables before ingestion.
  • Pre-aggregation constraints: Pre-aggregating metric rollups during ingestion reduces raw granularity, making it difficult to answer ad-hoc questions on unindexed dimensions later.
  • Maintenance overhead: Managing stateful distributed clusters of segment runners, historical nodes, and ZooKeeper or metadata brokers requires dedicated operational maintenance compared to serverless warehouses.
PreviousNext