The short answer
Enterprise data integration pipelines frequently extract data from on-premises transactional databases (such as Oracle, SAP, or Microsoft SQL Server) located behind corporate enterprise firewalls. Because cloud-hosted integration runtimes cannot reach private corporate IP addresses directly, Azure Data Factory relies on the Self-Hosted Integration Runtime (SHIR).
Communication model: Outbound HTTPS only
A major security advantage of the SHIR is its network communication topology.
Data engineers do not open inbound firewall ports into corporate data centers. Instead, the SHIR agent software (installed on Windows Server VMs inside the corporate private network) initiates all connections as outbound HTTPS traffic over port 443 to the Azure Data Factory cloud gateway service. The runtime pulls scheduled pipeline tasks from the cloud control plane and executes extraction jobs locally.
Architecting for high availability and scale
Deploying a single SHIR node creates a critical single point of failure: if the host VM crashes or reboots for OS patching, all scheduled production pipelines fail.
To achieve production high availability:
- Form a multi-node cluster by installing the SHIR software on up to 4 virtual machines.
- Register all nodes against the same ADF integration runtime authentication key.
- The nodes automatically operate in an active-active cluster: ADF distributes concurrent pipeline copy activities across all available nodes. If one VM fails, remaining cluster nodes automatically assume pending tasks without interrupting pipelines.
- Configure node concurrency limits based on available VM CPU and RAM to prevent pipeline memory exhaustion.