Handling corrupt and malformed records

When Spark reads JSON or CSV with a schema and a row does not fit, what happens depends on the read mode. There are three: PERMISSIVE, DROPMALFORMED and FAILFAST.

The three modes

spark.read.schema(schema).option("mode", "PERMISSIVE").json(path)

What each mode does:

• PERMISSIVE (the default): sets the fields it cannot parse to NULL, and if your schema includes a column for it, puts the raw text of the bad record there. The job carries on.
• DROPMALFORMED: silently throws away bad rows. Fast and quiet, which also makes it risky, because you lose data without a trace.
• FAILFAST: the job fails at the first bad record. Good for small, trusted feeds where bad data means a bug upstream.

Capturing the bad rows

In PERMISSIVE mode, the raw record goes into a column named _corrupt_record by default. For that to work, the column must be part of your explicit schema:

schema = StructType([
    StructField("order_id", LongType()),
    StructField("amount", DoubleType()),
    StructField("_corrupt_record", StringType()),
])
df = spark.read.schema(schema).json(path)
bad  = df.filter(F.col("_corrupt_record").isNotNull())
good = df.filter(F.col("_corrupt_record").isNull()).drop("_corrupt_record")

If you let Spark infer the schema, the column only appears when it finds bad records, and it is easy to miss. Spark also refuses queries that reference only the corrupt column on the raw file. Cache the DataFrame first, or include another column in the query.

Databricks also offers a badRecordsPath option that writes bad records to a folder for you.

What a good pipeline does

Do not just drop bad rows. Send them to a quarantine table together with the file name and the load time. Count them per run and alert when the rate goes above a threshold, for example more than 0.5 percent. A jump in bad records usually means that the producer changed something, and you want to know on the day it happens.

Choose the mode by risk. For a financial feed that must be complete, FAILFAST or PERMISSIVE with quarantine. For noisy clickstream logs, PERMISSIVE with a rate alert is normal.