The short answer
Serverless extract-transform-load () micro-pipelines often rely on AWS Lambda functions to extract incremental records from transactional relational databases (such as Amazon RDS PostgreSQL or Aurora). Because production databases reside within private VPC subnets with no public internet ingress, Lambda must be attached to the VPC.
Hyperplane Elastic Network Interfaces
Historically, connecting Lambda to a VPC caused cold starts lasting 10 to 30 seconds because AWS created an Elastic Network Interface (ENI) for every invocation.
AWS modernized this architecture with Hyperplane ENIs. AWS pre-provisions managed network adapters in your specified private subnets. Multiple concurrent Lambda executions share these managed Hyperplane interfaces, reducing VPC cold start overhead to under 1 second.
The internet and API access challenge
When a Lambda function runs inside a VPC, it loses default access to the public internet and public AWS service endpoints.
If your ETL function needs to read from private RDS and write extracted Parquet files to Amazon S3 while fetching database credentials from AWS Secrets Manager:
- Option 1: NAT Gateway: Route private subnet egress through an AWS NAT Gateway in a public subnet out to an Internet Gateway. While functional, NAT Gateways incur hourly base charges and data processing fees (check current pricing).
- Option 2: VPC Endpoints: Provision a Gateway VPC Endpoint for Amazon S3 (which is completely free) and Interface VPC Endpoints (AWS PrivateLink) for AWS Secrets Manager and CloudWatch Logs.
Private Subnet (VPC) +-------------------------------------------------------+ | Lambda Function ---> Hyperplane ENI ---> RDS Database | | | | | v | | Gateway Endpoint (S3) Interface Endpoint (Secrets) | +-------------------------------------------------------+
Using VPC endpoints keeps all data transmission entirely within the AWS global private network backbone, improving pipeline security posture while eliminating NAT Gateway data transfer fees.