Home

45-day sprint

The 45-day data engineer plan

Start Day 1

Be ready to take a real pipeline problem in a DE1 or DE2 interview, design it, code it and defend it. One coherent stack instead of 20 tools at 20 percent.

Practice as you go

The days below name the topic pages that train each skill. Open one to watch the idea move, then try its problems.

What you get

  • 45 guided daysA lesson, a lab and drills every day. Days 1 to 7 are free.
  • Real code in the browserSQL, Python and PySpark run in the editor and check your answer.
  • 3 capstone projectsBuilt end to end, ready for your GitHub and your CV.
  • 1,455+ problems and questionsCoding problems you run in the editor, interview questions with answers, and mock rounds that score you.
  • Revision built inReview days, spaced revision and a journal you can export on Day 45.
  • A Day 45 testDraw a full pipeline and defend it like you would in the interview.
Your plan rhythmOn pace

45-day intensive track

1 curriculum day daily (about 2.5 to 3.5 hours)

Day 0 diagnostic
Your progress
0 / 45(0%)
Next up
Day 1: SQL Foundations
Open Day 1 →
Projected finish
2027-01-03
Based on current 45-day pacing
Interview countdown
Optional, shown only to you

SQL Masterclass

Days 1 to 7Calendar days 1 to 7

SQL & Warehousing track →

  1. SELECT, WHERE, ORDER BY, DISTINCT, LIMIT, aliases, NULL, CASE, CAST, basic functions, why the database executes clauses in a fixed logical order

    Practice:GROUP BY

  2. GROUP BY, HAVING, COUNT, COUNT DISTINCT, SUM, AVG, MIN/MAX

    Practice:GROUP BY

  3. INNER, LEFT, RIGHT, FULL, CROSS, SELF JOIN

    Practice:JoinsSelf joins

  4. ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, SUM OVER, AVG OVER, PARTITION BY, ORDER BY

    Practice:Window functionsDeduplication

  5. CTEs, recursive CTE concept, subqueries, correlated subqueries, EXISTS, UNION, UNION ALL, INTERSECT, EXCEPT

    Practice:CTEsDeduplication

  6. indexes, query plans, EXPLAIN, EXPLAIN ANALYZE, partitioning, clustering concepts, predicate pushdown, avoiding SELECT *, joins and cardinality, materialized views

    Practice:JoinsWindow functions

Python for Data Engineering

Days 8 to 12Calendar days 8 to 12

Core Python track →Pandas track →

  1. data types, lists & dicts, hash sets, loops, functions, comprehensions, csv.DictReader

    Practice:StringsDictionariesArrays

  2. modules, exception handling, logging levels, file descriptors, JSON & CSV, env vars

    Practice:DictionariesRecursionSorting

  3. Day 10Pandas

    Pro

    read_csv, read_json, merge, groupby, vectorization, missing values, datetime, aggregation

    Practice:GROUP BYDictionaries

  4. config management, structured logging, custom exceptions, retries & backoff, idempotency, pytest

  5. REST extraction, schema validation, dead-letter quarantine, pure transforms, idempotent upsert, pytest

Data Engineering Fundamentals

Days 13 to 17Calendar days 13 to 17

SQL & Warehousing track →

  1. ETL vs ELT, batch vs streaming, micro-batching, medallion stages, raw immutable storage, replay recovery

  2. OLTP vs OLAP, fact tables, dimension tables, star vs snowflake, surrogate vs natural keys, declaring grain

  3. SCD Type 0 (Freeze), SCD Type 1 (Replace), SCD Type 2 (Version), valid_from & valid_to, is_current flag, point-in-time joins

  4. lake vs warehouse vs lakehouse, Medallion architecture, Parquet vs CSV, Hive partitioning, partition pruning, small files compaction

  5. medallion pipeline, bronze standardization, silver cleaning & dedup, gold star schema, SCD2 dimension update, reconciliation checks

Cloud

Days 18 to 22Calendar days 18 to 22

Cloud Platforms track →

  1. IAM roles & policies, STS temporary tokens, least privilege, S3 object model, cost safety runbooks, CloudWatch alarms

  2. S3 prefixes vs directories, Medallion zones (Raw/Silver/Gold), Hive partitioning, partition pruning, lifecycle tiering, compaction

  3. S3 + Glue + Athena flow, Glue Data Catalog, external tables & SerDes, Athena scan pricing ($5/TB, check current pricing), partition pruning, schema evolution

  4. Redshift MPP architecture, DISTSTYLE (KEY, ALL, EVEN), Sort Keys & Zone Maps, columnar compression, COPY from S3, atomic staging merges

  5. complete AWS platform flow, high-watermark extraction, S3 lakehouse staging, Glue catalog registration, Redshift COPY & merge, reconciliation

Spark / PySpark

Days 23 to 28Calendar days 23 to 28

PySpark track →

  1. horizontal scaling, driver vs executor, Catalyst DAG, transformations vs actions, narrow vs wide stages, fault tolerance lineage

    Practice:Aggregation

  2. lazy evaluation, the 9 core DataFrame ops, predicate pushdown, bitwise filtering (&, |), avoiding withColumn loops, window deduplication

    Practice:AggregationJoins

  3. partition boundaries, narrow vs wide transformations, shuffle write & fetch, repartition vs coalesce, coalesce(1) single-core hazard, partition sizing

    Practice:JoinsWindow functions

  4. BroadcastHashJoin (<100MB), partition pruning, Parquet predicate pushdown, when to cache vs unpersist, data skew key salting, small files coalesce

    Practice:Joins

  5. data contracts & early validation, string & enum sanitation, windowed latest-record deduplication, join cardinality diagnostics, pre-write circuit breakers, idempotent writes

    Practice:AggregationWindow functions

  6. Spark execution mental model, Spark UI triage framework, shuffle mechanics & skew, broadcast trade-offs, repartition vs coalesce, executor failure triage

    Practice:Window functionsJoins

Airflow + Orchestration

Days 29 to 32Calendar days 29 to 32

Orchestration track →

  1. DAGs as directed acyclic graphs, Scheduler vs Webserver vs Executor vs Worker, Task state lifecycle & cascades, Operators vs Sensors vs TaskFlow API, Bitshift dependency chaining (>>), XCom mechanics & metadata DB hazards

  2. Logical date vs Execution date vs Data interval, catchup=False vs catchup=True, CLI backfills & idempotent rerun safety, Sensor poke vs reschedule mode, Worker slot starvation prevention, Retries with exponential backoff & jitter, Avoiding top-level Variable.get() / Connection.get() DB blasts

  3. Separation of concerns: Airflow orchestrates, Spark computes, Why running PySpark code inside Airflow workers crashes worker pods, SparkSubmitOperator vs EMR / Dataproc / Synapse / Databricks operators, Deferrable operators & asynchronous triggers, S3/ADLS/GCS staging handoff: Extract → Raw → Spark → Curated → Warehouse COPY/MERGE, Atomic partition replacement & staging cleanup

  4. The 6 classic production orchestration failures, API rate limits (429) & exponential jitter backoff, Cloud storage eventual consistency & missing partition races, Source schema drift & dead-letter quarantine, Database connection pool exhaustion & connection pooling, Spark executor OOM retry cascade prevention, Metadata DB lock contention triage

dbt + Data Quality

Days 33 to 35Calendar days 33 to 35

dbt & Analytics track →Data Quality track →

  1. dbt architecture & compilation to native SQL, Sources, staging models & column renaming, ref() DAG generation & lineage, Generic tests (not_null, unique, accepted_values, relationships), dbt macros & Jinja templating, SCD Type 2 snapshots (check vs timestamp strategy)

  2. Defense in depth: Pre-load vs In-flight vs Post-load verification, Column invariants (uniqueness, referential integrity, range bounds), Row count anomaly detection & statistical deviation (Z-score), Freshness SLAs & silent pipeline stalls, Circuit breakers: quarantine vs pipeline abort, Data contract enforcement

  3. Full refresh vs incremental processing trade-offs, Watermark tracking & high-watermark state stores, Lookback windows for late-arriving records, Atomic MERGE & upsert mechanics (is_incremental() macro), Unique key deduplication before warehouse write, Handling hard deletes and tombstones

Project #1 - E-Commerce Data Platform

Days 36 to 38Calendar days 36 to 38
  1. Transactional database batch extraction, High-watermark state persistence, Partitioned Bronze raw lake staging, Schema validation & payload checksums, REST API event extraction, Idempotent extraction chunking

  2. Medallion curation: Bronze raw to Silver cleaned to Gold aggregates, Windowed deduplication by primary key and updated_at timestamp, Data contract casting & null substitution policies, Broadcast dimension enrichment, Gold daily revenue fact tables, Atomic S3 partition swaps

  3. End-to-end Airflow DAG tying extraction, Spark, and dbt together, Warehouse atomic MERGE / COPY ingestion gates, dbt data tests as blocking deployment gates, SLA monitoring & alert routing, Complete system architecture verbal defense, Cost & performance trade-off justification

Streaming + CDC

Days 39 to 40Calendar days 39 to 40

Streaming track →

  1. Distributed commit log architecture, Brokers, topics, partitions & segments, Message keys & deterministic partition routing, Producers: acks (0, 1, all) & idempotence, Consumer groups, rebalancing & offset commit semantics, Lag monitoring & consumer scale limits

  2. Change Data Capture (CDC) via database write-ahead log (WAL / binlog), Debezium connector architecture & event envelopes (before, after, op), Out-of-order events & watermark event-time processing, Deduplication & exactly-once processing guarantees, Streaming mutations and tombstones, Changelog table compaction

Job-ready engineering

Days 41 to 43Calendar days 41 to 43

System Design track →Infra Practices track →

  1. Junior pipeline design 6-step framework, Throughput, storage & compute capacity sizing, Decoupling ingestion, processing, and analytical serving, Batch vs streaming trade-off decisions, Simple GB/day storage & network calculations

  2. Git trunk-based development & feature branching, Data pipeline CI/CD with automated testing stages, Docker containerization for reproducible environments, Secrets management & environment variables, Infrastructure as Code principles, Observability: Metrics, logs, traces, and SLAs

  3. The classic production incident drills, Scenario 1: Pipeline takes 3 hours instead of 20 mins, Scenario 2: Spark job sudden executor OOM crash, Scenario 3: Financial revenue metric jumps unexpectedly, Scenario 4: Pipeline reports success but destination table is empty, Scenario 5: Duplicate records appear in downstream mart, Scenario 6: Upstream source introduces unannounced schema drift

Portfolio and final interview

Days 44 to 45Calendar days 44 to 45
  1. Project #1 portfolio packaging, Professional README authoring and architecture diagrams, Data dictionaries and schema documentation, Passing CI testing verification, Honest resume bullet drafting without buzzwords

  2. The 5 rounds of junior data engineering hiring, Round 1: SQL live coding (45 min), Round 2: Python data structures & core algorithms (45 min), Round 3: Data modeling & Spark concepts (45 min), Round 4: Pipeline system design (45 min), Round 5: Project walkthrough & behavioral defense (30 min)

Days 7, 14, 21, 28, 35, 42 are review days. Nothing new is taught, you revisit what you have built.

The 3 capstone projects

Build all 3 on one domain so your story stays coherent in the interview.

  1. AProject #1: Local Docker Medallion Pipeline

    Core PythonSQL & WarehousingPandasPySparkdbt & Analytics

  2. BReal-Time Customer Behavior Platform

    StreamingPySpark

  3. CCustomer 360 / CDC Platform

    OrchestrationSQL & Warehousing

See all projects

Best value: save 25%

Pro 3 months: ₹899 for 90 days₹1,197

Save ₹298 (25% off). Just ₹300 a month.

Days 1 to 7 are free. The pass covers all 45 days, with 45 days of buffer for revision and missed days. It includes full access, so the drill bank, mock interviews and readiness are included.

One-time pass. No auto-renew, no autopay mandate.

45-Day Plan reviews & ratings

Finished a few lessons? Tell us what worked and what was confusing. Reviews are read by hand and help decide what gets rewritten next.

FAQ

Do I have to finish in exactly 45 days?
No. It is a pace, not a rigid deadline. If you are working full-time, follow the 90-day working professional track at 1 to 1.5 hours per day. Consistent daily practice beats cramming.
How much time do I need each day?
We recommend choosing one of two realistic rhythms based on your schedule: Intensive Track (~2.5 to 3.5 hours/day over 45 days: 1 hour learning concepts and running the lab plus 1.5 to 2.5 hours hands-on practice), or Working Professional Track (~1 to 1.5 hours/day spread over 90 days).
Will this get me a data engineer job in 45 days?
No plan can promise that. Finishing every day here means you can write the transforms, the DAG habits, the tests, and the design argument. It does not publish a GitHub repo or sit the interview for you. See the note on experience below.
I already know SQL. Can I skip ahead?
Yes. Start wherever your actual gap is and use this page for pacing the rest.
Does LakeBench run real AWS, Kafka, and Airflow?
No, and it should not pretend to. SQL and Python run for real in the editor. Cloud, orchestration, and streaming are taught as mocked SDKs and brokers in Python. Docker, Kubernetes, Terraform, and a live AWS account are outside this tab. See what maps where on the roadmap page.