Serialization: Java, Kryo, Tungsten and Arrow

Where serialization happens

Why DataFrames mostly do not care

A DataFrame row in Tungsten format is a compact byte layout: a null bitmap, fixed-width slots for numbers, and offsets to variable-length values like strings. Operators read fields straight from those bytes, and shuffling a row means copying its bytes. There are no JVM objects per row to create, serialize or garbage-collect. This is a big part of why DataFrames beat RDDs of objects, and why changing spark.serializer rarely changes a pure DataFrame job.

Java vs Kryo for RDDs

spark = (SparkSession.builder
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .config("spark.kryo.registrationRequired", "true")     # fail fast on unregistered classes
    .config("spark.kryo.classesToRegister", "com.shop.Order,com.shop.Customer")
    .getOrCreate())

A "Kryo serialization failed: Buffer overflow" error means one object is larger than spark.kryoserializer.buffer.max (64 MB); raise it, or avoid huge single records such as giant collected arrays.

Task not serializable

The classic error: a function shipped to executorsexecutor: A worker process on a cluster machine that runs tasks and holds data in memory. Learn more → captures an object that cannot be serialized, such as a database connection, a client, or (in Scala) the enclosing class. Create such objects on the executor, once per partitionpartition: A chunk of a DataFrame's rows. Spark processes each partition as one task, so partitions decide how much work runs in parallel. Learn more →:

def save_partition(rows):
    conn = make_connection()          # created on the executor
    for row in rows:
        conn.write(row.asDict())
    conn.close()

orders.foreachPartition(save_partition)

Arrow for Python

A row-at-a-time Python UDF pickles rows in the JVM, unpickles them in Python, calls your function once per row, and pickles the results back. Apache Arrow moves whole column batches in a format both sides read without conversion.

Common mistakes

• ✗Switching to Kryo to speed up a DataFrame job — DataFrames use Tungsten binary rows; the serializer setting barely matters there.
• ✗Capturing a connection or client in a closure — Task not serializable, or one connection per row. Create it per partition on the executor.
• ✗toPandas without Arrow on large results — Slow row-by-row conversion on top of collecting everything to the driverdriver: The process that runs your program, plans the work and sends tasks to executors. Learn more →.