BigData New Added 12 September 2026

Spark: org.apache.spark.SparkException: Task not serializable

A closure sent to the executors dragged an unserialisable object along with it. Referencing one field of an enclosing class captures the whole instance, so a lambda that only needs a string can pull in a database connection, a logger or the SparkSession itself.

Quick fix

Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.

Quick fix

// Fails: the lambda captures `this` to reach the field
class Job(spark: SparkSession) {
  val prefix = "id-"
  def run(ds: Dataset[String]) = ds.map(s => prefix + s)
}

// Works: copy the value into a local before the closure is built
def run(ds: Dataset[String]) = {
  val p = prefix
  ds.map(s => p + s)
}

// Connections belong inside the partition, never captured from the driver
df.foreachPartition { rows =>
  val conn = DriverManager.getConnection(url)
  try rows.foreach(r => insert(conn, r)) finally conn.close()
}

# The trace names the offending field once you ask for it
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer

How to diagnose BigData errors

Distributed data errors are rarely about the line of code that threw them. An executor OOM, a failed checkpoint or a stuck consumer group are usually symptoms of data skew, insufficient partitioning, or a downstream system applying backpressure. The diagnostic instinct that pays off here is to look at the distribution of work across tasks before looking at the exception.

If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.

Open the Spark UI (or Flink dashboard) and sort tasks by duration and shuffle read size. If one task is an order of magnitude larger than the median, you have skew, not a memory shortage.
Check the driver log and at least one executor log. The exception the driver reports is often a downstream consequence of the first executor failure.
For streaming, measure consumer lag over time rather than at a point (kafka-consumer-groups --describe). Lag that grows linearly means throughput, lag that spikes and recovers means a poison message or a GC pause.
Confirm whether the job failed or was killed. YARN, Kubernetes and Databricks all kill containers that exceed memory limits, and the resulting message looks like a crash rather than an eviction.
Before increasing memory, try increasing partitions. Doubling spark.sql.shuffle.partitions is cheaper and more often correct than doubling executor memory.

Tools worth reaching for

Spark UI
kafka-consumer-groups
Airflow task logs
EXPLAIN / query profile

Authoritative references

Primary documentation for this error, worth reading before applying any fix in production.

spark.apache.org

Related BigData errors

See all 11 BigData errors →

Browse other categories

Something missing or wrong?

This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.