Polars is a DataFrame library written in Rust, built on the Apache Arrow memory format. It is multi-threaded by default, can optimise whole queries before running them, and handles data larger than memory through a streaming engine. Data engineers like it because it is fast and uses less memory than pandas on the same job.
Expressions instead of row functions
import polars as pl
result = (
pl.scan_parquet("orders/*.parquet")
.filter(pl.col("status") == "paid")
.group_by("country")
.agg(pl.col("amount").sum().alias("revenue"))
.sort("revenue", descending=True)
.collect()
)You describe columns and operations with expressions (pl.col(...)). Polars runs them in its Rust engine in parallel, so you seldom need a Python apply.
Lazy mode and the optimiser
scan_parquet and .lazy() build a query plan and do nothing until .collect(). Before running, Polars optimises it, much like a SQL engine: predicate pushdown (apply filters while reading the file), projection pushdown (read only the columns used), and merging steps. In the example above, only status, country and amount are read, and rows that are not paid are discarded early. explain() shows the plan.
Beyond memory
With .collect(engine="streaming") (the exact option name differs by version), Polars processes data in batches, so a file larger than RAM can be handled for many query shapes.
Differences from pandas
- No index. Rows are identified by position, which removes a class of alignment bugs.
- Stricter types, with real null values (not NaN for everything).
- Different API:
with_columns,group_by, expressions. Code does not port line for line. - Results are immutable-style: operations return new frames.
Trade-offs
The ecosystem is smaller. Some libraries that expect pandas need a conversion (.to_pandas(), often cheap because of Arrow). The API has changed across releases while the library matured, so pin the version. Team familiarity matters: if everyone knows pandas, the learning cost is real.
How to answer
Say what it is (Rust, Arrow, parallel, lazy optimiser), give one benefit you would measure (time and memory on your own job), and mention the cost (smaller ecosystem). Interviewers like the comparison to a query engine.