Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. External tables, BigLake and Omni

Snowflake, BigQuery & Databricks · BigQuery

External tables, BigLake and Omni

Mediumwarehouses-32
bigqueryexternal-tablesbiglakeomnifederation

Question

How can BigQuery query data that isn't stored in BigQuery?

Solution

BigQuery can query data that lives outside its own storage. The options are external tables, BigLake tables, federated queries to other databases, and BigQuery Omni for other clouds.

External tables

An external table is a definition over files in Cloud Storage (Parquet, ORC, Avro, CSV, JSON), or data in Bigtable or Google Drive. BigQuery reads the files at query time. No data is loaded, so it is useful for quick access to a data lake, or for files you will load later. It is usually slower than native tables, because there are no BigQuery-managed statistics and layout for the data. There is no time travel on external data, and caching and pruning options are more limited.

BigLake tables

BigLake improves on external tables. It adds fine-grained security (row and column policies) over lake files, so users can be given access to a table without access to the bucket. It supports metadata caching for faster query planning on big directory trees, and it can work with Apache Iceberg tables. It also lets engines like Spark access the same data with the same governance through connectors.

CREATE EXTERNAL TABLE lake.events
WITH CONNECTION `us.my_conn`
OPTIONS (format = 'PARQUET', uris = ['gs://my-bucket/events/*.parquet']);

Federated queries

EXTERNAL_QUERY sends a query to Cloud SQL, Spanner or AlloyDB and joins the result with BigQuery data. It is handy for small lookups against an operational database, without building a pipeline. It puts load on the source database and is not meant for big scans.

BigQuery Omni

Runs BigQuery's engine next to data in AWS S3 or Azure storage, so you can query it without moving it. Results can be returned to BigQuery, or exported.

Trade-offs

Query external data when it saves you building a pipeline, when data must stay where it is, or when it is queried rarely. Load it into native tables when the queries are frequent, need good performance, or you want features like time travel and clustering. A lot of teams do both: land the raw files in a lake, and load the curated layers to native tables.

PreviousNext