IAM roles, execution policies, and least privilege for AWS data pipelines

Structure of an execution role

An IAM role consists of two distinct JSON policy documents that separate identity trust from resource permissions:

• Trust policy: Defines which trusted entity can assume the role. For example, AWS Glue jobs use a trust policy with the service principal glue.amazonaws.com, allowing the Glue service to assume the execution role on your behalf.
• Permission policy: Defines the exact actions and resources the role is permitted to interact with, specifying Action, Effect, and Resource elements.

AWS Service Principal (glue.amazonaws.com)
  | Assumes Role via Trust Policy (sts:AssumeRole)
  v
IAM Role (GluePipelineExecutionRole)
  | Evaluates Attached Permission Policy
  v
Target Resource: s3://lake-prod/curated/* (s3:PutObject allowed)

Implementing least privilege in practice

Beginner pipelines often assign the overly permissive AdministratorAccess or broad wildcard s3:* policy. Production data pipelines enforce least privilege through specific controls:

• Restrict S3 actions by prefix, granting read access only to s3://my-bucket/raw/* and write access only to s3://my-bucket/curated/*.
• Scope database permissions so pipeline roles can execute specific Athena queries without granting rights to drop production databases.
• Use sts:AssumeRole with external identifiers for cross-account access, allowing a central data platform to ingest data from separate production accounts securely.