Azure Entra ID and RBAC for data pipelines

Managed identities eliminate static keys

Hardcoding account keys or connection strings in configuration files creates severe credential leakage risks. Azure provides managed identities that assign an automatic identity in Entra ID to Azure resources such as Azure Data Factory, Azure Databricks, or virtual machines:

• System-assigned managed identity: Tied directly to the lifecycle of the pipeline service instance. When the Azure Data Factory instance is deleted, Azure deletes its identity automatically.
• User-assigned managed identity: Created as an independent Azure resource that multiple services can share, ideal for pipelines spanning multiple runtimes.

Control plane versus data plane roles

A common mistake in Azure data engineering is confusing management access with data access:

• The management role Contributor allows engineers to modify storage account networking rules or delete the account, but grants zero permission to read blob contents.
• The data plane role Storage Blob Data Contributor grants read, write, and delete permissions on blob containers and files.
• The read-only role Storage Blob Data Reader allows query engines like Synapse Serverless SQL to read Parquet datasets without write permissions.

Pipeline Service (ADF / Databricks)
  | Authenticates via Managed Identity (OAuth token)
  v
Microsoft Entra ID
  | Evaluates Azure RBAC Role Assignment
  v
ADLS Gen2 Container (Storage Blob Data Contributor)

Data pipelines should always bind data plane roles to managed identities at the container scope rather than assigning full subscription access.