Batch & Streaming · Batch Processing

What is MapReduce?

EasyMapReducebatchHadoopshuffle

Interview question

What is MapReduce?

How to answer

The short answer

MapReduce is a programming model for large batch jobs on a cluster. You write two main steps:

  1. 1Map: turn each input record into intermediate key/value pairs (in parallel)
  2. 2Reduce: group by key and aggregate/combine values for that key

The framework handles splitting files, shipping data, sorting by key, and restarting failed tasks.

Output
Input splits
   | map | map | map |
        \  |  /
      shuffle + sort by key
        /  |  \
   | reduce | reduce |
        \  |  /
         output

Classic word-count mental model

  • Map: (word, 1) for each token
  • : all counts for hello go to the same reducer
  • Reduce: sum the 1s

Interview tip: MapReduce taught the industry "scale by partitioning." Spark kept the idea but kept data in memory across steps to avoid writing every stage to disk.