45-day sprint
The 45-day data engineer plan
Be ready to take a real pipeline problem in a DE1 or DE2 interview, design it, code it and defend it. One coherent stack instead of 20 tools at 20 percent.
- 45 days, 11 phases
- Days 1 to 7 free
- View the full roadmap →
- Set up your tools →
- Glossary →
Practice as you go
The days below name the topic pages that train each skill. Open one to watch the idea move, then try its problems.
What you get
- 45 guided daysA lesson, a lab and drills every day. Days 1 to 7 are free.
- Real code in the browserSQL, Python and PySpark run in the editor and check your answer.
- 3 capstone projectsBuilt end to end, ready for your GitHub and your CV.
- 1,455+ problems and questionsCoding problems you run in the editor, interview questions with answers, and mock rounds that score you.
- Revision built inReview days, spaced revision and a journal you can export on Day 45.
- A Day 45 testDraw a full pipeline and defend it like you would in the interview.
45-day intensive track
1 curriculum day daily (about 2.5 to 3.5 hours)
SQL Masterclass
Days 1 to 7Calendar days 1 to 7Day 1SQL Foundations
FreeSELECT, WHERE, ORDER BY, DISTINCT, LIMIT, aliases, NULL, CASE, CAST, basic functions, why the database executes clauses in a fixed logical order
Practice:GROUP BY
Day 4Window Functions
FreeROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, SUM OVER, AVG OVER, PARTITION BY, ORDER BY
Practice:Window functionsDeduplication
Day 5Advanced SQL
FreeCTEs, recursive CTE concept, subqueries, correlated subqueries, EXISTS, UNION, UNION ALL, INTERSECT, EXCEPT
Practice:CTEsDeduplication
Day 6SQL Performance
Freeindexes, query plans, EXPLAIN, EXPLAIN ANALYZE, partitioning, clustering concepts, predicate pushdown, avoiding SELECT *, joins and cardinality, materialized views
Practice:JoinsWindow functions
Practice:JoinsWindow functionsDeduplication
Python for Data Engineering
Days 8 to 12Calendar days 8 to 12data types, lists & dicts, hash sets, loops, functions, comprehensions, csv.DictReader
Practice:StringsDictionariesArrays
Day 9Real Python
Promodules, exception handling, logging levels, file descriptors, JSON & CSV, env vars
Practice:DictionariesRecursionSorting
read_csv, read_json, merge, groupby, vectorization, missing values, datetime, aggregation
Practice:GROUP BYDictionaries
Day 11Production Python
Proconfig management, structured logging, custom exceptions, retries & backoff, idempotency, pytest
Day 12Python Interview Day
ProREST extraction, schema validation, dead-letter quarantine, pure transforms, idempotent upsert, pytest
Data Engineering Fundamentals
Days 13 to 17Calendar days 13 to 17ETL vs ELT, batch vs streaming, micro-batching, medallion stages, raw immutable storage, replay recovery
OLTP vs OLAP, fact tables, dimension tables, star vs snowflake, surrogate vs natural keys, declaring grain
SCD Type 0 (Freeze), SCD Type 1 (Replace), SCD Type 2 (Version), valid_from & valid_to, is_current flag, point-in-time joins
lake vs warehouse vs lakehouse, Medallion architecture, Parquet vs CSV, Hive partitioning, partition pruning, small files compaction
medallion pipeline, bronze standardization, silver cleaning & dedup, gold star schema, SCD2 dimension update, reconciliation checks
Cloud
Days 18 to 22Calendar days 18 to 22IAM roles & policies, STS temporary tokens, least privilege, S3 object model, cost safety runbooks, CloudWatch alarms
S3 prefixes vs directories, Medallion zones (Raw/Silver/Gold), Hive partitioning, partition pruning, lifecycle tiering, compaction
S3 + Glue + Athena flow, Glue Data Catalog, external tables & SerDes, Athena scan pricing ($5/TB, check current pricing), partition pruning, schema evolution
Redshift MPP architecture, DISTSTYLE (KEY, ALL, EVEN), Sort Keys & Zone Maps, columnar compression, COPY from S3, atomic staging merges
complete AWS platform flow, high-watermark extraction, S3 lakehouse staging, Glue catalog registration, Redshift COPY & merge, reconciliation
Spark / PySpark
Days 23 to 28Calendar days 23 to 28Day 23Spark Fundamentals
Prohorizontal scaling, driver vs executor, Catalyst DAG, transformations vs actions, narrow vs wide stages, fault tolerance lineage
Practice:Aggregation
Day 24PySpark DataFrames
Prolazy evaluation, the 9 core DataFrame ops, predicate pushdown, bitwise filtering (&, |), avoiding withColumn loops, window deduplication
Practice:AggregationJoins
Day 25Spark Transformations
Propartition boundaries, narrow vs wide transformations, shuffle write & fetch, repartition vs coalesce, coalesce(1) single-core hazard, partition sizing
Practice:JoinsWindow functions
Day 26Spark Performance
ProBroadcastHashJoin (<100MB), partition pruning, Parquet predicate pushdown, when to cache vs unpersist, data skew key salting, small files coalesce
Practice:Joins
data contracts & early validation, string & enum sanitation, windowed latest-record deduplication, join cardinality diagnostics, pre-write circuit breakers, idempotent writes
Practice:AggregationWindow functions
Day 28Spark Interview Day
ProSpark execution mental model, Spark UI triage framework, shuffle mechanics & skew, broadcast trade-offs, repartition vs coalesce, executor failure triage
Practice:Window functionsJoins
Airflow + Orchestration
Days 29 to 32Calendar days 29 to 32Day 29Airflow Fundamentals
ProDAGs as directed acyclic graphs, Scheduler vs Webserver vs Executor vs Worker, Task state lifecycle & cascades, Operators vs Sensors vs TaskFlow API, Bitshift dependency chaining (>>), XCom mechanics & metadata DB hazards
Day 30Production Airflow
ProLogical date vs Execution date vs Data interval, catchup=False vs catchup=True, CLI backfills & idempotent rerun safety, Sensor poke vs reschedule mode, Worker slot starvation prevention, Retries with exponential backoff & jitter, Avoiding top-level Variable.get() / Connection.get() DB blasts
Separation of concerns: Airflow orchestrates, Spark computes, Why running PySpark code inside Airflow workers crashes worker pods, SparkSubmitOperator vs EMR / Dataproc / Synapse / Databricks operators, Deferrable operators & asynchronous triggers, S3/ADLS/GCS staging handoff: Extract → Raw → Spark → Curated → Warehouse COPY/MERGE, Atomic partition replacement & staging cleanup
The 6 classic production orchestration failures, API rate limits (429) & exponential jitter backoff, Cloud storage eventual consistency & missing partition races, Source schema drift & dead-letter quarantine, Database connection pool exhaustion & connection pooling, Spark executor OOM retry cascade prevention, Metadata DB lock contention triage
dbt + Data Quality
Days 33 to 35Calendar days 33 to 35Day 33dbt Core & Modeling
Prodbt architecture & compilation to native SQL, Sources, staging models & column renaming, ref() DAG generation & lineage, Generic tests (not_null, unique, accepted_values, relationships), dbt macros & Jinja templating, SCD Type 2 snapshots (check vs timestamp strategy)
Day 34Data Quality
ProDefense in depth: Pre-load vs In-flight vs Post-load verification, Column invariants (uniqueness, referential integrity, range bounds), Row count anomaly detection & statistical deviation (Z-score), Freshness SLAs & silent pipeline stalls, Circuit breakers: quarantine vs pipeline abort, Data contract enforcement
Full refresh vs incremental processing trade-offs, Watermark tracking & high-watermark state stores, Lookback windows for late-arriving records, Atomic MERGE & upsert mechanics (is_incremental() macro), Unique key deduplication before warehouse write, Handling hard deletes and tombstones
Project #1 - E-Commerce Data Platform
Days 36 to 38Calendar days 36 to 38Transactional database batch extraction, High-watermark state persistence, Partitioned Bronze raw lake staging, Schema validation & payload checksums, REST API event extraction, Idempotent extraction chunking
Medallion curation: Bronze raw to Silver cleaned to Gold aggregates, Windowed deduplication by primary key and updated_at timestamp, Data contract casting & null substitution policies, Broadcast dimension enrichment, Gold daily revenue fact tables, Atomic S3 partition swaps
End-to-end Airflow DAG tying extraction, Spark, and dbt together, Warehouse atomic MERGE / COPY ingestion gates, dbt data tests as blocking deployment gates, SLA monitoring & alert routing, Complete system architecture verbal defense, Cost & performance trade-off justification
Streaming + CDC
Days 39 to 40Calendar days 39 to 40Day 39Streaming basics
ProDistributed commit log architecture, Brokers, topics, partitions & segments, Message keys & deterministic partition routing, Producers: acks (0, 1, all) & idempotence, Consumer groups, rebalancing & offset commit semantics, Lag monitoring & consumer scale limits
Change Data Capture (CDC) via database write-ahead log (WAL / binlog), Debezium connector architecture & event envelopes (before, after, op), Out-of-order events & watermark event-time processing, Deduplication & exactly-once processing guarantees, Streaming mutations and tombstones, Changelog table compaction
Job-ready engineering
Days 41 to 43Calendar days 41 to 43Junior pipeline design 6-step framework, Throughput, storage & compute capacity sizing, Decoupling ingestion, processing, and analytical serving, Batch vs streaming trade-off decisions, Simple GB/day storage & network calculations
Day 42Shipping code safely
ProGit trunk-based development & feature branching, Data pipeline CI/CD with automated testing stages, Docker containerization for reproducible environments, Secrets management & environment variables, Infrastructure as Code principles, Observability: Metrics, logs, traces, and SLAs
The classic production incident drills, Scenario 1: Pipeline takes 3 hours instead of 20 mins, Scenario 2: Spark job sudden executor OOM crash, Scenario 3: Financial revenue metric jumps unexpectedly, Scenario 4: Pipeline reports success but destination table is empty, Scenario 5: Duplicate records appear in downstream mart, Scenario 6: Upstream source introduces unannounced schema drift
Portfolio and final interview
Days 44 to 45Calendar days 44 to 45Day 44Ship your portfolio
ProProject #1 portfolio packaging, Professional README authoring and architecture diagrams, Data dictionaries and schema documentation, Passing CI testing verification, Honest resume bullet drafting without buzzwords
The 5 rounds of junior data engineering hiring, Round 1: SQL live coding (45 min), Round 2: Python data structures & core algorithms (45 min), Round 3: Data modeling & Spark concepts (45 min), Round 4: Pipeline system design (45 min), Round 5: Project walkthrough & behavioral defense (30 min)
Days 7, 14, 21, 28, 35, 42 are review days. Nothing new is taught, you revisit what you have built.
The 3 capstone projects
Build all 3 on one domain so your story stays coherent in the interview.
AProject #1: Local Docker Medallion Pipeline
CCustomer 360 / CDC Platform
Best value: save 25%
Pro 3 months: ₹899 for 90 days₹1,197
Save ₹298 (25% off). Just ₹300 a month.
Days 1 to 7 are free. The pass covers all 45 days, with 45 days of buffer for revision and missed days. It includes full access, so the drill bank, mock interviews and readiness are included.
One-time pass. No auto-renew, no autopay mandate.
45-Day Plan reviews & ratings
Finished a few lessons? Tell us what worked and what was confusing. Reviews are read by hand and help decide what gets rewritten next.
FAQ
- Do I have to finish in exactly 45 days?
- No. It is a pace, not a rigid deadline. If you are working full-time, follow the 90-day working professional track at 1 to 1.5 hours per day. Consistent daily practice beats cramming.
- How much time do I need each day?
- We recommend choosing one of two realistic rhythms based on your schedule: Intensive Track (~2.5 to 3.5 hours/day over 45 days: 1 hour learning concepts and running the lab plus 1.5 to 2.5 hours hands-on practice), or Working Professional Track (~1 to 1.5 hours/day spread over 90 days).
- Will this get me a data engineer job in 45 days?
- No plan can promise that. Finishing every day here means you can write the transforms, the DAG habits, the tests, and the design argument. It does not publish a GitHub repo or sit the interview for you. See the note on experience below.
- I already know SQL. Can I skip ahead?
- Yes. Start wherever your actual gap is and use this page for pacing the rest.
- Does LakeBench run real AWS, Kafka, and Airflow?
- No, and it should not pretend to. SQL and Python run for real in the editor. Cloud, orchestration, and streaming are taught as mocked SDKs and brokers in Python. Docker, Kubernetes, Terraform, and a live AWS account are outside this tab. See what maps where on the roadmap page.