Pipelines & scenarios interview questions for data engineers
End-to-end design, retries, CDC, skew, shuffle, monitoring, and security. 45 questions with full answers, written for freshers. All topics
Core Pipeline Concepts (2)
Design & Scenarios (1)
System Design Questions (12)
- Design a real-time clickstream analytics pipeline
- Design a daily batch pipeline from an OLTP database
- Design a CDC pipeline to the lakehouse
- Design an ingestion framework for 200 sources
- Design a pipeline for IoT sensor data
- Design a data warehouse for customer support tickets
- Design a feature pipeline for ML
- Design a GDPR/DPDP-compliant user data platform
- Design a metrics layer for company KPIs
- Migrate an on-prem Hadoop/Hive warehouse to the cloud
- Design a pipeline that ingests files from partners via SFTP
- Design a reverse ETL flow
Production Scenarios (12)
- Upstream data is late
- Pipeline ran twice and doubled the data
- Source adds a new column without telling you
- Job runtime keeps growing every week
- Backfill 2 years for a new column
- One bad record fails the whole batch
- API source returns inconsistent pages
- Timezone bug in daily partitions
- Warehouse costs spiked after a deploy
- Two pipelines writing to the same table
- Reprocess after a bug in transformation logic
- Process files that arrive out of order
Core Concepts Not Yet Covered (18)
- Push vs pull ingestion
- Staging area
- Atomic publish and table swaps
- Exactly-once vs effectively-once in batch
- Data pipeline testing pyramid
- Environments: dev, staging, prod for data
- Observability for pipelines: what to measure
- Alert fatigue
- Runbooks
- Rate of change: append-only vs mutable data
- Batch size and parallelism for loads
- Connection pooling in ETL jobs
- Data versioning
- Handling slowly arriving dimensions in pipelines
- Ingestion tools: Fivetran, Airbyte, custom
- Data pipeline vs ML pipeline
- Pipelines for LLM and RAG applications
- Vector databases