System design interview

Feature Store for ML (Batch + Real-Time)

HardLocked55 min read

Design a feature store for batch training (millions of vectors) and real-time inference (<10ms), with point-in-time correctness.

system-designfeature-storemlstreamingpoint-in-time

Start here

The problem in plain words

A model that predicts whether a customer will buy needs inputs such as "average spend over the last 30 days". Those inputs are called features. During training, a data scientist computes this feature in a notebook from a big historical table. When the model goes live, an engineer writes the same feature again in the app, so that it can be computed in a few milliseconds.

Now there are two versions of one idea, written by two people. One of them counts the window a little differently. Worse, the training table may already include a purchase from today, so the model quietly peeks at the answer while learning. In the lab the model looks brilliant. In production it behaves worse, and nobody knows why.

A feature store is a shared shelf where each feature is defined once, and the same definition produces the history used for training and the latest value used for live predictions.

Words you will meet

Feature
One input value for a model, such as a customer's average spend over 30 days.
Entity
The thing a feature describes, such as one customer or one product. Features are looked up by an entity id.
Training vs inference
Training is teaching the model from history. Inference is using the finished model to predict for a live request.
Label
The correct answer the model learns from, such as "did buy" or "did not buy".
Offline store
Big, cheap tables of feature values with timestamps, used to build training data.
Online store
A fast key-value database, such as Redis or DynamoDB, holding only the latest feature values for instant lookups.
Point-in-time join
Attaching to each training example the feature values as they were known at that moment, not as they are today.
Leakage
When training data contains information from after the moment of the prediction. The model cheats and looks better than it is.
Training-serving skew
A gap between how a feature is computed for training and how it is computed live.
p99 latency
The time that 99 out of 100 requests finish within. A p99 of 10 ms means almost every lookup is faster than 10 ms.
Two versions of one feature
trainingliveworseTraining notebookSQL on a historicaldumpModellearns and goes liveProduction codeits own version
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.The notebook computes "spend in the last 30 days" for training. The table already contains today's purchase, so the model sees the future.
  2. 2.In production, separate code computes the same feature with a slightly different window and different default for missing data.
  3. 3.The model is fed numbers that do not mean the same thing as in training. Accuracy drops and nobody can find the bug.

The numbers, as things you can picture

  • Under 10 ms at p99A live prediction waits for features. If almost every lookup is not under 10 ms, the product feels slow.
  • Thousands of requests a secondLive lookups come in constantly, so the online store has to be built for speed, not for big scans.
  • Dozens to hundreds of features per modelEach prediction asks for many values at once, so keep each value small.
  • Millions to billions of rows of historyThe offline store is large and is read in big batches for training.

What the interviewer wants

Design a feature store that serves training data in batch and live values in under 10 ms, using one definition for both, and explain how you stop the model from peeking at the future.

Interview framing

You are designing a feature store for machine learning in a senior-leaning DE interview. About 55 minutes. The scoring center is training-serving skew and point-in-time correctness, not only Redis plus Snowflake.

The prompt:

Design a feature store that serves batch training (millions of historical feature vectors) and real-time inference (fetch one feature vector in under 10 ms), with point-in-time correctness so training does not leak future information.

Background from first principles

A feature is an input signal to a model: average spend last 30 days, clicks in the last hour, account age. Training needs large historical tables. Inference needs a single entity lookup in milliseconds.

If training computes average spend last 30 days from a warehouse dump that already includes today's purchase, the model learns the future. Offline metrics look great. Online performance collapses. That failure mode is training-serving skew (and label leakage when the label itself sneaks into features).

A feature store exists so one definition of a feature materializes into two serving surfaces: offline (lakehouse) and online (low-latency key-value), without two conflicting SQL dialects living in different teams' notebooks.

Contrast with analytics dashboards. Dashboards can tolerate approximate tiles and human judgment. Models amplify silent mismatches. That is why this problem is stricter about consistency.

Expanded problem and constraints

  • Offline: read millions to billions of rows for training dataset generation.
  • Online: get(entity_keys) under about 10 ms p99 in the inference region.
  • Point-in-time correct joins for labels at time T.
  • Backfill after feature logic changes.
  • Monitor training-serving skew (compare offline recompute vs online values).
  • Replayable compute from raw events or tables.

What good looks like

You tell a skew story first. You propose two stores, one definition. You explain point-in-time joins with a tiny timeline example. You make the online path a KV lookup, not a warehouse query. You discuss missing-feature behavior and versioning.

You also separate batch-daily features from true streaming features instead of pretending everything is real-time on day one.

Clarifying questions

  • Entity keys: user, device, listing, or multiple entity types?
  • Do online features need time travel or only latest values?
  • Batch-only features (T+1) vs true streaming features (seconds)?
  • Who authors feature definitions: ML or DE?
  • Inference QPS and payload size per vector?
  • Multi-region inference?
  • What happens when a feature is missing at score time?

Scale and estimation prompts

  • Offline storage is cheap at lakehouse scale; online keeps latest features per entity (plus short history only if required).
  • Inference QPS may be thousands; latency dominates. Redis/Dynamo/Cassandra-style stores.
  • Feature count per model: dozens to hundreds. Watch online memory and network.
  • Estimate vector size: 200 features x 8 bytes is small; nested maps and huge embeddings change the story.

Out of scope for v1

  • Using the warehouse as the online inference path.
  • Guaranteeing identical streaming freshness for every feature on day one (mix batch and stream deliberately).
  • Full time travel for every online key unless product requires it.
  • Building a complete managed platform clone if a thin in-house spine meets the latency and PIT needs.
  • Training the model itself (focus on feature serving and correctness).

Why DE owns part of this

Feature stores sit between data platform and ML platform. Data engineers bring correctness under time, backfills, and reliable materialization. ML engineers bring feature intent and model constraints. The interview rewards candidates who can speak both languages without dissolving ownership.

Extra product constraints

Confirm whether fraud models need sub-second streaming features while growth models can wait for daily batch. Confirm embedding features and their sizes. Confirm whether point-in-time correctness is required only for training or also for batch scoring jobs that simulate past decisions.

What interviewers listen for

Skew story, PIT join, two stores one definition, sub-10ms KV path, versioning, and monitoring. Logo soup without those themes fails.