To process files as they land in Amazon S3, configure S3 Event Notifications or Amazon EventBridge rules to emit object-created events into an intermediate Amazon SQS queue. Placing a queue between S3 and downstream processing buffers traffic spikes, absorbs backpressure, and handles automatic retries with dead-letter isolation. Because serverless functions have strict memory and runtime limits, small files can be parsed directly in AWS Lambda while multi-gigabyte files hand off to AWS Glue, EMR, or Step Functions.
Decoupled event routing with message queues
Directly invoking compute functions from S3 bucket notifications creates operational vulnerabilities during sudden landing bursts.
S3 Object Created ---> EventBridge / SQS Queue ---> Lambda Worker
| (Failure after retries)
+---> Dead-Letter Queue (DLQ)A resilient event-driven architecture includes several defensive design layers:
- Route S3 notifications through Amazon SQS rather than triggering Lambda functions directly. If thousands of files land simultaneously, SQS buffers the events so downstream consumers scale at a controlled concurrency rate without getting throttled.
- Configure dead-letter queues on the SQS queue. If a malformed file crashes worker logic, the message retries up to a configured threshold before routing to the dead-letter queue, preventing consumer blockage.
- Handle duplicate and out-of-order events inside consumer code. S3 events guarantee at-least-once delivery, so pipelines must record processed object keys or rely on idempotent database merges to prevent duplicated records.
Compute boundaries and hand-off patterns
Engineers match compute engines to payload dimensions:
- AWS Lambda provides rapid execution for lightweight payloads, but it operates with hard technical boundaries: a fifteen-minute execution limit and ten gigabytes of memory.
- For small JSON, CSV, or parquet files that process in seconds, Lambda parses records and writes directly to target tables.
- For massive multi-gigabyte files or complex multi-step transformations, Lambda acts solely as an orchestrator. It receives the S3 notification, extracts metadata, and triggers an AWS Glue ETL job, an Amazon EMR step, or an AWS Step Functions workflow to coordinate heavy processing.