Pub/Sub to BigQuery direct streaming ingestion without Dataflow

While Dataflow provides streaming transformations, windowing, and enrichment, deploying Dataflow merely to pass JSON payloads directly from Pub/Sub into BigQuery introduces unnecessary infrastructure costs, worker provisioning delays, and operational pipeline monitoring.

Pub/Sub BigQuery Subscriptions

Google Cloud provides BigQuery Subscriptions, allowing Pub/Sub topics to stream messages directly into BigQuery tables with zero intermediate compute infrastructure.

Event Producers (IoT, Apps, Microservices)
                    |
                    v
          Google Cloud Pub/Sub Topic
                    |
                    v Direct ingestion (No Dataflow workers!)
       BigQuery Subscription (Managed by Pub/Sub)
                    |
                    v Writes via BigQuery Storage Write API
          Target BigQuery Table

Architectural features and capabilities

• Zero compute management: You provision no VM clusters, container pods, or Dataflow workers; Google manages streaming write capacity internally.
• Cost reduction: Eliminates hourly Dataflow worker VM and persistent disk charges; you pay only standard Pub/Sub throughput fees and BigQuery ingestion rates (check current pricing).
• Automatic schema mapping: When enabled, Pub/Sub maps JSON message fields directly to target BigQuery table columns. Unmatched fields or metadata attributes (such as publish timestamp and message ID) can be written to specialized metadata columns.
• Dead-letter queuing: If an incoming message cannot be parsed due to a schema mismatch, Pub/Sub automatically routes the malformed record to a designated dead-letter topic without stalling ingestion.

When Dataflow is still required

Use BigQuery Subscriptions for straightforward ingestion into raw landing tables. Use Cloud Dataflow when streaming records require complex windowed aggregations, joining against external databases, or real-time PII tokenization before landing in storage.