Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Batch loading vs streaming cost in BigQuery

Snowflake, BigQuery & Databricks · BigQuery

Batch loading vs streaming cost in BigQuery

Mediumwarehouses-27
bigquerybatch-loadstreamingcostload-jobs

Question

Why are batch loads into BigQuery free but streaming isn't?

Solution

Batch loads from Cloud Storage into BigQuery do not cost anything for the load itself, because load jobs use a shared pool of slots that Google provides at no charge. Streaming ingestion is billed by the volume of data you send. That is the whole reason for the difference: a load job uses spare shared capacity on its own schedule, and streaming needs always-on capacity to make rows available right away.

Batch load

bq load --source_format=PARQUET analytics.orders gs://my-bucket/orders/2025-03-01/*.parquet

You pay for storing the files in Cloud Storage while they exist, and for BigQuery storage afterwards, but not for the load. A load job is also atomic. It either adds all the rows or none, so a failed job leaves no half-loaded table. That makes retries simple.

There are quotas, such as a limit on load jobs per table per day, and limits on file counts and sizes. Check the current numbers, because they are the reason to avoid launching thousands of tiny load jobs.

Streaming

The Storage Write API and the older streaming insert charge per GB ingested. At a few GB a day that is small. At terabytes a day it becomes real money.

The middle path

Often "near real time" does not mean seconds. If a dashboard is fine with 5 or 10 minutes of lag, write small files to Cloud Storage and load them every few minutes with a load job. Each load covers many rows, stays free, and is atomic. This micro-batch pattern is common, and Pub/Sub plus a small Dataflow or Cloud Run job writing files fits it well.

Choosing

  • Seconds of latency needed: streaming, accepting the cost.
  • Minutes acceptable: micro-batch loads.
  • Hourly or daily: plain batch loads.

Say that you would ask what freshness the business needs before picking, since cost grows quickly as latency shrinks.

PreviousNext