The short answer
Data engineering teams often deploy heavy distributed compute clusters (such as Amazon EMR or AWS Glue Spark jobs) to process batches containing hundreds of thousands of independent files. However, maintaining Spark clusters for simple file transformations introduces cluster startup latency, driver overhead, and compute idle costs.
AWS Step Functions Distributed Map provides a serverless orchestration pattern for high-concurrency .
Architecture of Distributed Map
The Distributed Map state extends Step Functions execution capabilities:
- It can read an Amazon S3 bucket prefix, an S3 inventory list, or a large or CSV file containing millions of records.
- It automatically splits the items into batches.
- It launches up to 10,000 parallel execution iterations concurrently.
- Each iteration can trigger a lightweight child workflow, such as invoking an AWS Lambda function or running an AWS Fargate container task.
Step Functions Workflow +-------------------------------------------------------------+ | Distributed Map State | | Reads S3 Inventory List (500,000 objects) | | | | +---------------+ +---------------+ +---------------+ | | | Child Task 1 | | Child Task 2 | | Child Task N | | | | (Lambda ETL) | | (Lambda ETL) | | (Lambda ETL) | | | +---------------+ +---------------+ +---------------+ | | (Scales dynamically up to 10,000 parallel workers) | +-------------------------------------------------------------+
When to choose Distributed Map over Spark
Use Distributed Map when:
- Each item can be processed independently (embarrassingly parallel tasks, such as resizing images, parsing individual JSON logs, or calling external validation APIs).
- Workloads are irregular or bursty: you pay only for the exact state transitions and compute seconds consumed, scaling down to zero immediately.
Choose Spark on EMR or Glue when:
- Workloads require complex cross-record , multi-table joins, window aggregations, or heavy distributed state management across .