Read a file bigger than memory

Never load the whole file. A 50 GB CSV does not fit in 16 GB of RAM, and pandas usually needs several times the file size in memory. So you process it in pieces, or hand it to an engine that streams for you. Start with what question you need answered, because that decides which approach is cheap.

Option 1: stream with the standard library

The csv module reads one row at a time and holds almost nothing in memory.

import csv
from collections import Counter

counts = Counter()
with open("orders.csv", newline="") as f:
    for row in csv.DictReader(f):
        counts[row["country"]] += 1

It is simple and slow-ish for 50 GB, but it works on any laptop.

Option 2: pandas in chunks

import pandas as pd

totals = {}
for chunk in pd.read_csv(
    "orders.csv",
    usecols=["country", "amount"],
    dtype={"country": "category", "amount": "float32"},
    chunksize=1_000_000,
):
    part = chunk.groupby("country")["amount"].sum()
    for k, v in part.items():
        totals[k] = totals.get(k, 0) + v

Each chunk is a normal DataFrame of a million rows. You aggregate it, keep only the small result, and move on. Two details save a lot of memory: usecols skips columns you do not need, and explicit dtype avoids pandas guessing wide types. This pattern works for sums, counts, min and max, where partial results combine easily. Medians and exact distinct counts do not combine so simply.

Option 3: convert once, then query fast

A CSV is slow to read and carries no types. Read it in chunks once and write Parquet, which is compressed and columnar, then query that many times.

The practical answer: DuckDB or Polars

import duckdb
duckdb.sql("""
  SELECT country, SUM(amount) AS revenue
  FROM read_csv('orders.csv')
  GROUP BY country
""").show()

DuckDB streams the file, uses all cores, spills to disk if needed, and gives you SQL. Polars scan_csv with lazy evaluation does something similar. For one-off analysis on a laptop, this is what most engineers would actually do.

What to avoid

df = pd.read_csv("orders.csv") on the full file. It either runs out of memory after minutes, or swaps the machine to a crawl.

If the job must run every day, a single laptop is the wrong place anyway. Move it to a Spark job or a warehouse load, and keep the laptop approach for exploration.