Foundations
PySpark End-to-End Master Study Notebook
Spark architecture, DataFrames, joins and tuning, from first job to a working pipeline.
- Handwritten-style
- Diagrams and tables
- Worked examples
Who this notebook is for
Learners moving from Python or SQL to Spark, and engineers preparing for PySpark interviews who need the why behind the code.
What you will be able to do
- Explain how Spark splits a job into stages and tasks
- Write DataFrame and Spark SQL code, including joins and window functions
- Reduce shuffles and fix skew with partitioning, broadcast joins and AQE
- Read and write formats and Delta tables, and build a simple streaming job
What's inside
Pick a chapter and the notebook opens right there.
- Apache Spark & PySpark FoundationsDriver, Executors, Py4J daemon, SparkSession, Client vs Cluster, DAG lazy evaluationStart here
- Resilient Distributed Datasets (RDDs)5 RDD properties, map/flatMap/filter, reduceByKey vs groupByKey, lineage & cache/checkpointStart here
- PySpark DataFrames & Spark SQLCatalyst distributed tables, StructType schemas, CSV/Parquet/JDBC I/O, column ops & temp viewsStart here
- Advanced Column Ops & Complex TypesMath/string functions, dates & timestamps, Arrays (explode), Structs/Maps & Higher-Order functionsStart here
- Aggregations, Grouping & Window FunctionsgroupBy & multi-agg, rollup/cube/grouping sets, row_number, rank, lead/lag & running totalsStart here
- Joins, Data Merging & Physical StrategiesInner/outer/cross, left_semi vs left_anti, Broadcast Hash Join (BHJ) vs Sort Merge Join (SMJ)Start here
- User-Defined Functions (UDFs) & VectorizationPython UDF overhead, Arrow-backed Pandas UDFs (@pandas_udf), applyInPandas & Koalas APIStart here
- Catalyst Optimizer & Execution InternalsAnalysis, Logical Optimization (pushdown/pruning), Cost-Based Optimizer, Janino Whole-Stage CodeGen & explain()Start here
- Memory Management & Project TungstenJVM Heap pools, Execution vs Storage dynamic borrowing, Off-heap binary format, G1GC tuning & OOM fixesStart here
- Partitioning, Shuffling & Data DistributionDefault split size, narrow vs wide dependencies, shuffle write/read, repartition vs coalesce & bucketingStart here
- Performance Tuning & AQEAdaptive Query Execution, dynamic partition coalescing, runtime BHJ switch, skew splitting & saltingStart here
- Data Sources, Formats & I/O OptimizationParquet/ORC file skipping, predicate pushdown, JSON parsing modes, JDBC parallelism & compactionStart here
- Delta Lake & Lakehouse with PySparkACID transactions, _delta_log protocol, MERGE upserts, Time Travel history, Z-ORDER & schema evolutionStart here
- Structured Streaming FoundationsUnbounded append table, micro-batching, Kafka integration, output modes (Append/Complete/Update) & checkpointsStart here
- Advanced Streaming & Stateful ProcessingTumbling/sliding windows, watermarking late data, stream-stream joins, deduplication & stateful processingStart here
- PySpark Machine Learning (MLlib)Feature engineering (VectorAssembler, Scalers, StringIndexer), ML Pipelines, distributed classifiers & tuningStart here
- Cluster Deployment, Submissions & Monitoringspark-submit flags, K8s vs YARN operators, Spark UI deep dive, event logs & dynamic allocationStart here
- Testing, CI/CD & Best PracticesPyTest session fixtures, Chispa DataFrame equality assertions, secrets decoupling, static typing & CI automationStart here
- Real-World Production ArchitecturesClickstream sessionization, real-time fraud detection, CDC upsert pipelines, GDPR purges & feature storesStart here
- FAANG & Staff DE Interview ScenariosHigh-frequency coding rounds, Catalyst grill, live OOM triage, straggler task remediation & sizing cheat cardStart here
- Modern Spark 3.4+ & Cloud-NativeSpark Connect (gRPC), Delta UniForm (Iceberg), K8s Celeborn shuffle & RocksDB stateStart here
- Gap-Fill SupplementAuto Loader & declarative pipelines, speculation & storage- partitioned joins, stats/observe, SCD2 MERGE, quality gatesStart here
Practice this
Put it to work on LakeBench.
Read the guide first
Short explainers that pair with this notebook.
- What is Apache Spark? A beginner's guide to how it works
- PySpark production problems: data skew explained with visuals
- PySpark collect_list vs collect_set, and explode vs explode_outer
- spark.sql.shuffle.partitions explained: what it does and how to set it
- Scenario-based PySpark interview questions, with runnable answers
Common questions
- Do I need to know Spark already?
- No. It starts with what Spark is and how it runs, then moves to DataFrames. Knowing basic Python and SQL first will make it easier.
- Does it cover Spark performance questions?
- Yes. There are chapters on partitioning and shuffling, memory, performance tuning with AQE, and interview scenarios for senior roles.
- Can I read a sample before I commit?
- Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.
