Cloud · Azure in Depth

Error handling in Azure Data Factory: retries, failure paths and pipeline status

azureMediumazureadferror-handlingretriesmonitoringdependencies

Interview question

How do you handle failures in an ADF pipeline, and why can a pipeline show Succeeded even when an activity failed?

How to answer

The short answer

ADF gives you two separate tools: retries for transient problems and dependency paths for deciding what happens next.

Retries. Each activity has a retry count, a retry interval and a timeout. A Copy activity that fails because the source was briefly unreachable can try again after 30 seconds without any pipeline logic. Retries suit temporary faults, not wrong credentials or a missing table.

Dependency paths. An arrow leaving an activity can be one of four types:

PathRuns when
Upon successthe activity succeeded (the default)
Upon failurethe activity failed
Upon completionthe activity finished, either way
Upon skipthe activity did not run

At most one path is taken per run, and Upon completion cannot sit beside Upon success or Upon failure.

Why the pipeline status can surprise you. ADF decides the overall result by looking at the leaf activities, and the pipeline succeeds only if all of them succeed. That gives three common designs:

  • Try-catch: only an Upon failure path. If the main activity fails and the handler succeeds, the pipeline shows Succeeded. The failure is hidden from monitoring unless the handler raises an alert.
  • Do-if-else: both Upon success and Upon failure paths. If the main activity fails, the pipeline shows Failed even when the handler runs fine. Use this when operations must see the failure.
  • Do-if-skip-else: adds a dummy Upon skip activity, so a handled failure shows Succeeded.

A typical production setup uses Upon failure to log the error to a table and send a notification, then lets the pipeline fail so the run is visible, and sets up Azure Monitor alerts on failed pipeline runs.

Interview tip: the question behind this question is "how will anyone know it failed?" Always say what raises the alert, and be clear whether the pipeline should end as Failed.