Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

Read CSV with explicit schema

Mediumcsvschemaiodataframe

Interview question

How do you read a CSV in PySpark with an explicit schema, and why is that better than inferSchema?

How to answer

The short answer

In production, prefer an explicit schema when you read. Inference scans data (extra cost), can guess wrong types, and makes pipelines brittle when a dirty row appears.

Explicit schema example

Python
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, IntegerType, DateType

schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("user_id", IntegerType(), True),
    StructField("amount", DoubleType(), True),
    StructField("order_date", DateType(), True),
    StructField("status", StringType(), True),
])

orders = (
    spark.read
    .option("header", True)
    .option("mode", "PERMISSIVE")  # or FAILFAST / DROPMALFORMED
    .schema(schema)
    .csv("s3://lake/raw/orders/")
)

Why not inferSchema alone

Python
# Fine for exploration only
df = spark.read.option("header", True).option("inferSchema", True).csv(path)
  • Inference may promote IDs to integers incorrectly, or keep numbers as strings inconsistently.
  • contracts belong in code or a table metastore.

Diagram

Output
CSV files --> reader (header + schema) --> DataFrame with known types
                 \-- bad rows --> _corrupt_record (PERMISSIVE)

Interview add-on

Discuss mode (PERMISSIVE/FAILFAST), date formats, multiline /CSV options, and enforcing schema at the bronze-to-silver boundary.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill