Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Private networking for data pipelines

Cloud · Cloud Architecture Basics

Private networking for data pipelines

Hardcloud-42
networkingvpcsecurityprivate-linkprivate-endpoints

Question

How do you keep data pipeline traffic off the public internet?

Solution

To isolate data pipeline traffic from the public internet, deploy databases and compute nodes exclusively on private subnets with private IP addresses and communicate with cloud services using private VPC endpoints. Use AWS PrivateLink or GCP Private Service Connect to access managed APIs like Amazon S3 and Google BigQuery over private network backbones rather than public internet routes. For Google Cloud compute without external IP addresses, enable Private Google Access so workers can resolve APIs, and deploy VPC Service Controls to prevent unauthorized data exfiltration.

Private endpoints and routing controls

Exposing database ports or pipeline infrastructure to public IP ranges creates immediate security exposure.

Private Subnet (Compute VMs / DBs)
         |
    VPC Endpoint / Private Service Connect (Private IP)
         |
    Cloud Backbone (No Public Internet) ---> S3 / BigQuery / Cloud Storage

Enterprise networks enforce multiple layers of traffic isolation:

  • Database instances like Amazon RDS and Google Cloud SQL are provisioned without public IP addresses, accessible only by compute nodes within the same VPC or peered networks.
  • Configure VPC endpoints (Gateway endpoints for S3 and DynamoDB in AWS) and Private Service Connect in GCP. These endpoints route API calls over private provider backbones instead of resolving public internet gateways.
  • When worker nodes need to download third-party software packages or access external partner APIs, route outbound traffic through a managed Cloud NAT gateway with strict egress firewall rules.
  • Worker clusters deployed on subnets without public IPs, such as Google Cloud Dataflow or Dataproc nodes, require Private Google Access enabled on the subnet to fetch Google Cloud APIs, BigQuery storage endpoints, and container images.

Data exfiltration boundaries

Network isolation protects against external penetration, but platforms must also guard against insider threats and unauthorized data movement:

  • VPC Service Controls in Google Cloud create security perimeters around projects, blocking BigQuery and Cloud Storage API calls from exfiltrating data outside approved network boundaries.
  • Even if a developer possesses valid IAM credentials, the security perimeter denies attempts to copy data from an enterprise BigQuery dataset to a personal storage bucket outside the perimeter.
PreviousNext