Cloud · Azure in Depth

CI/CD for Azure Data Factory: Git, ARM templates and environments

azureHardazureadfci-cdarm-templategitdevops

Interview question

How do you promote an Azure Data Factory from dev to test to production with CI/CD?

How to answer

The short answer

ADF stores every pipeline, dataset and linked service as . CI/CD means moving that definition between factories (dev, test, prod) without anyone clicking in the production portal.

The setup

  • Only the development factory is connected to Git. Test and production factories are never edited by hand; they only receive deployments.
  • Developers work on feature branches, debug their changes, and open a pull request into the collaboration branch (often main). Pushing straight to main is blocked.
  • ADF packages its resources as an ARM template, which is Azure's JSON format for describing resources to deploy. Deployment means applying that template to the target factory.

Two ways to produce the ARM template

  1. 1The Publish button. A person opens the ADF user interface and clicks Publish. ADF writes the ARM template to the adf_publish branch, and a release pipeline deploys from there. This works, but it needs a manual click.
  2. 2Automated publish. A build pipeline runs on every merge to main, using the npm package @microsoft/azure-data-factory-utilities. Its validate command runs the same checks as "Validate all", and export produces the ARM template as a build artifact. The release pipeline deploys that artifact. Both approaches are supported.
Output
feature branch --PR--> main --build pipeline: validate + export--> ARM template artifact
                                   |
                                   +--release--> test factory --approval--> prod factory

Environment differences are handled with ARM template parameters: storage account names, linked service endpoints, and similar values change per stage. Keep secrets in a separate Key Vault per environment with the same secret names, so connection strings do not need overriding.

Things that catch teams out

  • Triggers must be stopped before deployment and restarted afterwards. A pre- and post-deployment script does this, and the newer packaging can limit it to triggers that changed.
  • ADF does not support publishing a single pipeline. Publishing includes every change in the factory, so use a hotfix process for urgent fixes.
  • Integration runtime names and types should match across stages.

Microsoft's current guidance also points new projects toward Data Factory in Microsoft Fabric, which has its own deployment model.