Service accounts and workload identity

Data pipelines should authenticate to cloud services using managed identities such as AWS IAM roles or GCP service accounts attached directly to compute environments, eliminating long-lived secret key files. For external environments like GitHub Actions or Kubernetes clusters, teams implement workload identity federation to exchange short-lived OpenID Connect tokens for temporary cloud access. Every pipeline must operate under least-privilege permissions scoped to specific storage prefixes and datasets, and credentials must never be committed to source code or baked into container images.

IAM roles and short-lived credentials

Relying on hardcoded API tokens or downloaded JSON service account keys is one of the most common security vulnerabilities in data engineering.

Compute Worker (EC2 / GKE / Cloud Run)
       |
  Metadata API ---> Requests Temporary Token (OAuth / STS)
       |
  Cloud Services (S3 / BigQuery) ---> Validates Token & Applies Scoped IAM

Production pipelines implement credential management through automated identity attachment:

• Attach IAM roles or service accounts directly to the compute resources executing the pipeline, such as Amazon EC2, Amazon ECS, Google Kubernetes Engine, or Google Cloud Run.
• The compute instance queries its local metadata service to retrieve temporary, self-rotating credentials automatically, meaning application code never handles secret keys.
• Never bake credentials into Docker container images or push secret configuration files into version control systems.
• Audit pipeline activity continuously through services like AWS CloudTrail and Google Cloud Audit Logs to identify unused permissions or anomalous access patterns.

Federated identity and least privilege

Modern data pipelines often run across hybrid environments and continuous deployment runners:

• Workload identity federation allows external platforms like GitHub Actions or on-premises Kubernetes pods to authenticate directly with cloud IAM using OpenID Connect (OIDC). The runner exchanges an ephemeral signed identity token for a temporary cloud credential without storing static secrets.
• Enforce strict least-privilege permissions for each pipeline. An ingestion job writing raw clickstream files needs put-object permissions only on the bronze bucket prefix, not full administrative access to the entire data platform.
• Separate development, staging, and production identities so a compromised pipeline in a testing environment cannot modify production analytical tables.