PySpark interview questions for data engineers
Spark architecture, DataFrames, shuffles, joins, and streaming. 56 questions with full answers, written for freshers. All topics
Core Concepts (2)
DataFrame API & I/O (1)
Caching & Reliability (1)
Execution Model (13)
- Job vs stage vs task
- Client mode vs cluster mode
- What spark-submit does
- Cluster managers
- map vs flatMap
- reduceByKey vs groupByKey
- Lineage and fault tolerance
- Speculative execution
- Dynamic allocation
- Accumulators and broadcast variables
- Project Tungsten and whole-stage codegen
- Serialization: Java vs Kryo
- How PySpark talks to the JVM
Memory, OOM & Tuning (9)
Joins & Data Layout (9)
DataFrame API in Practice (12)
- Window functions in PySpark
- Handling corrupt and malformed records
- Schema inference vs explicit schema
- Flatten nested JSON
- explode vs explode_outer vs posexplode
- collect_list, collect_set and array functions
- union vs unionByName
- Built-in functions before UDFs
- Reading from JDBC sources in parallel
- count() vs approx_count_distinct
- toPandas and collect safely
- Deduplicating with window vs dropDuplicates