The short answer
Hardcoded storage account access keys, database passwords, and connection strings in configuration files represent one of the most common security vulnerabilities in data engineering. If an engineer accidentally commits an access key to source control, the entire corporate data lake can be compromised.
Azure Managed Identities eliminate the need to store, manage, or rotate credentials in pipeline code.
System-assigned versus User-assigned identities
Microsoft Entra ID provides two varieties of managed identities:
- System-assigned managed identity: Tied directly to the lifecycle of an individual Azure resource (such as an Azure Data Factory instance, Synapse workspace, or Azure VM). When the resource is created, an identity is registered in Entra ID. If the resource is deleted, Azure automatically destroys the identity.
- User-assigned managed identity: Created as an independent Azure resource with its own lifecycle. Multiple services (for example, three distinct ADF factories or Dataproc clusters) can share the same user-assigned identity, simplifying permission governance across complex multi-service pipelines.
The authentication workflow
When an Azure Data Factory pipeline needs to read data from an ADLS Gen2 storage container:
- The data engineer assigns the
Storage Blob Data ContributorRBAC role directly to the ADF managed identity in Azure Portal or Terraform. - During pipeline execution, ADF requests a short-lived OAuth 2.0 security token directly from the local Azure Entra ID metadata endpoint.
- ADF presents the token to the storage service to authenticate the request.
No passwords or connection strings are ever created, stored in code, or maintained in Azure Key Vault. Token acquisition, encryption, and automatic rotation are handled entirely by Azure infrastructure.