Cloud
Google Cloud (GCP) Data Engineer
BigQuery, Cloud Storage, Dataflow, Pub/Sub and Composer, with interview questions.
- Handwritten-style
- Diagrams and tables
- Worked examples
Who this notebook is for
Engineers and candidates preparing for Google Cloud data roles who want BigQuery, streaming and orchestration explained together.
What you will be able to do
- Design BigQuery tables for speed and cost
- Build streaming pipelines with Pub/Sub and Dataflow, including windows and watermarks
- Orchestrate batch work with Composer and Dataproc
- Apply governance and sensitive data protection with Dataplex and DLP
What's inside
Pick a chapter and the notebook opens right there.
- Google Cloud Storage (GCS) & Storage ArchitectureFlat namespace, storage classes (Standard/Nearline/Coldline/Archive), lifecycle policies (OLM), dual-region, CMEK, Autoclass & BigLake integrationStart here
- BigQuery Architecture, Storage & IngestionDremel execution engine, Capacitor columnar format, On-Demand vs Editions/Reservations, bq load, Storage Write API streaming & multi-stream commitsStart here
- BigQuery Performance Optimization & Physical TuningPartitioning (ingestion, date, integer range), Clustering on high-cardinality keys, BI Engine in-memory acceleration, Materialized Views & slot contention triageStart here
- BigQuery Advanced Analytics: ML, Search & OmniBigQuery ML algorithms (linear, logistic, k-means, boost trees), Vector Search & embeddings, BigQuery Omni multi-cloud queries, Analytics Hub federated data exchangesStart here
- Cloud Pub/Sub & Real-Time Ingestion ArchitectureTopics & subscriptions (Push vs Pull), ordering keys, dead-letter topics (DLQ), exactly-once delivery, direct BigQuery & GCS streaming sinksStart here
- Cloud Dataflow & Apache Beam FoundationsUnified batch & streaming, Pipeline, PCollection, ParDo/DoFn, GroupByKey, CombinePerKey, Runner v2 & Streaming Engine offload architectureStart here
- Advanced Dataflow Streaming: Windows, Watermarks & StateFixed, sliding & session windows, event-time watermarks, trigger accumulators, stateful DoFn with StateSpec & Timers, and backpressure triageStart here
- Cloud Bigtable & Cloud Spanner: Operational Data StoresBigtable wide-column row key design, tablet splitting, garbage collection, Spanner globally distributed ACID, TrueTime API & Interleaved TablesStart here
- Cloud Dataproc & Cloud Composer: Orchestration & ComputeManaged Spark/Hadoop, Dataproc Serverless, Ephemeral clusters with Spot VMs, Cloud Composer 2 on Autopilot GKE, CeleryExecutor & DAG sensorsStart here
- Dataplex, Data Governance & SecurityLakehouse zones, metadata harvesting, policy tags, column masking, VPC Service Controls & perimeter securityStart here
- Datastream CDC & Real-Time Database ReplicationOracle/PostgreSQL redo log CDC, BigQuery CDC UPSERT via BigQuery CDC UPSERT via _CHANGE_TYPE _CHANGE_TYPE, zero-downtime database migration, zero-downtime database migrationStart here
- BigQuery Continuous Queries & Apache Iceberg / BigLakeReal-time SQL reverse-ETL, Apache Iceberg metadata snapshots, time travel & multi-engine query federationStart here
- Dataform & Modern Analytics EngineeringSQLX declarative pipelines, incremental materialization, unit testing, assertion contracts & release workflow compilationStart here
- Sensitive Data Protection (Cloud DLP) & Security150+ InfoType detectors, de-identification transforms (crypto-hash, bucketization, pseudonymization), audit findingsStart here
- Google Cloud FinOps & Cost OptimizationOn-Demand vs Editions, Slot Reservations & Idle Sharing, GCS Autoclass economics, Dataflow Streaming Engine offload, budget slashersStart here
- Hybrid, Multi-Cloud & Cross-Region NetworkingDedicated/Partner Interconnect, Cloud HA- VPN, Private Service Connect (PSC), Storage Transfer Service (STS), BigQuery OmniStart here
- Production Observability, SRE & SLA MonitoringLog Routers, BigQuery INFORMATION_SCHEMA INFORMATION_SCHEMA, Dataflow system lag & watermark delay, Pub/Sub DLQ, multi-, Dataflow system lag & watermark delay, Pub/Sub DLQ, multi- window burn rates, blameless RCAsStart here
- Staff Data Engineer FAANG GCP System DrillsYouTube 1B event sessionizer, Uber geospatial H3 Bigtable fleet matching, Spanner banking ledger, AdTech RTB BigLake feature store, $50k billing triageStart here
- Gap-Fill Supplement: Data Fusion, BigQuery Programmability & StoresCloud Data Fusion, BigQuery scripting/UDFs/remote functions/search indexes, Firestore, Memorystore, LookerStart here
Practice this
Put it to work on LakeBench.
Common questions
- Does it explain Apache Beam?
- Yes. Two chapters cover Dataflow and Beam, including windowing, watermarks and state for streaming jobs.
- Is BigQuery tuning included?
- Yes. A full chapter covers performance and physical tuning, and another covers ML, search and Omni.
- Can I read a sample before I commit?
- Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.
