Read Large CSV in Chunks

Pandas data engineering interview problem. Difficulty: beginner. Pattern: Aggregation. About 12 minutes. Free to practice.

Aggregate chunked frames without loading one giant CSV. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Some CSV files are too big to fit in memory, so pandas can read them in pieces called chunks with read_csv(..., chunksize=...). You handle one chunk at a time and never hold the whole file. In the editor, chunks is a list of three small DataFrames, which stand in for those pieces. Each has user_id and amount. Add up amount for each user_id across all the chunks, and return a DataFrame with the columns user_id and amount, sorted by user_id. Do the adding for each chunk first, and combine the small results at the end. Do not glue the raw chunks together before you add them up, because that would defeat the purpose of reading in chunks. Save the final table in result.

Requirements

• Return 1 row for each user_id.
• amount is the total of that user's amounts across all chunks.
• Return only the columns user_id and amount.
• Sort by user_id.
• Aggregate each chunk first, then combine the small results.

Constraints

• chunks is a list of 3 DataFrames with the columns user_id and amount.
• The same user can appear in more than one chunk, so one chunk is not enough to know a user's total.
• In a real file the full data would not fit in memory, but the per-user totals would.

Examples

Input: three chunks Output: user_id | amount 1 | 15.0 2 | 21.0 3 | 7.0 4 | 9.0 Chunk-local sums merge correctly.

Topics: lakebench, pandas, chunks, groupby.

More interview problems · All interview problems · Learn data engineering