DuckDB is an analytical database that runs inside your Python process, with no server to install or manage. It stores and processes data in columns, and you query it with SQL, including directly on Parquet and CSV files.
What it looks like
import duckdb
con = duckdb.connect() # in-memory, or duckdb.connect("local.db") for a file
con.sql("""
SELECT country, SUM(amount) AS revenue
FROM read_parquet('s3://bucket/orders/*.parquet')
GROUP BY country
ORDER BY revenue DESC
""").show()You can query a folder of Parquet files on disk, on S3 or on GCS (with the right extension and credentials), without loading them into a database first. It reads only the needed columns and row groups, and uses all CPU cores.
Talking to pandas, Polars and Arrow
DuckDB can query a pandas or Polars DataFrame in memory by name, and return results as a DataFrame, using Arrow so little or no copying happens.
con.sql("SELECT * FROM df WHERE amount > 100").df()Where it fits in a data engineering workflow
- Local development and tests: run the same SQL transformations against small sample files, without a warehouse and its cost.
- Small to medium transformations that fit on one machine (up to hundreds of GB of Parquet, thanks to its out-of-core abilities), as a fast alternative to Spark for that size.
- Exploring and profiling files, checking a Parquet file's contents with a single query.
- Data quality checks and reconciliation on files.
- Embedded analytics inside an application.
- With dbt: the dbt-duckdb adapter runs dbt models locally.
What it is not
It is not a server for many users. It is built for one process at a time writing (several can read a file read-only), so it does not replace a shared warehouse. It is not distributed either, so one machine's memory and disk are the limit.
How to answer
Call it "SQLite for analytics". Say it is columnar and vectorised, queries files directly, and gives you a fast, free tool for tests and mid-sized jobs. Add the limit: single machine, not multi-user.