The short answer
Amazon Kinesis Data Firehose provides a serverless ingestion service to deliver streaming records directly into destinations such as Amazon S3, Redshift, and OpenSearch without managing ingestion servers.
Configuring Firehose buffering parameters represents a critical architectural trade-off between streaming data freshness and downstream query efficiency.
Buffer size versus buffer interval
Firehose buffers incoming streaming records in memory before writing an aggregated object to Amazon S3. The buffer flushes whenever either of two thresholds is met first:
- Buffer size: Threshold ranging from 1 MB to 128 MB.
- Buffer interval: Threshold ranging from 60 seconds to 900 seconds (15 minutes).
If you configure a 60-second interval on a low-volume stream producing only 100 KB per minute, Firehose writes a tiny 100 KB file every minute. Over 1 month, this creates over 43,000 tiny files in S3, triggering the well-known in Spark and Athena.
Conversely, setting a 128 MB buffer size with a 900-second interval on high-volume streams ensures Firehose bundles data into healthy 128 MB blocks, which match Hadoop and Spark file splitting targets.
Built-in format conversion and compression
Rather than dumping raw files into S3 and scheduling hourly Glue jobs to convert them to Parquet, configure Firehose native features:
- conversion: Firehose integrates with AWS Glue schemas to convert incoming JSON streams into columnar Parquet format inline before writing to S3.
- Snappy compression: Applying Snappy compression reduces storage volume by 70% to 80% and lowers Athena scan costs.
- Dynamic partitioning: Firehose automatically parses attributes within the JSON payload (such as
customer_idorevent_date) and routes records into custom S3 key prefixes dynamically.