BigData New Added 15 October 2026

Spark: [CAST_INVALID_INPUT] The value 'abc' of the type "STRING" cannot be cast

Spark 4.0 turned ANSI SQL mode on by default. A cast that used to quietly return NULL for a malformed value now fails the whole query, and so do integer overflow and division by zero ([DIVIDE_BY_ZERO]). Jobs that ran for years start failing on the first dirty row after the upgrade, and the bad data was always there: Spark 3 was just hiding it as NULLs.

Quick fix

Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.

Quick fix

# Is ANSI mode on for this session?
spark-sql -e "SET spark.sql.ansi.enabled"

# Find the rows that do not parse
SELECT raw_col FROM events WHERE try_cast(raw_col AS INT) IS NULL AND raw_col IS NOT NULL;

# Fix: say explicitly that malformed input should become NULL
SELECT try_cast(raw_col AS INT) AS n FROM events;
SELECT try_divide(total, count) FROM stats;

# PySpark equivalent
# df.select(F.expr("try_cast(raw_col AS INT)"))

# Stopgap only: restore the Spark 3 behaviour for one job
spark-submit --conf spark.sql.ansi.enabled=false job.py

How to diagnose BigData errors

Distributed data errors are rarely about the line of code that threw them. An executor OOM, a failed checkpoint or a stuck consumer group are usually symptoms of data skew, insufficient partitioning, or a downstream system applying backpressure. The diagnostic instinct that pays off here is to look at the distribution of work across tasks before looking at the exception.

If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.

Open the Spark UI (or Flink dashboard) and sort tasks by duration and shuffle read size. If one task is an order of magnitude larger than the median, you have skew, not a memory shortage.
Check the driver log and at least one executor log. The exception the driver reports is often a downstream consequence of the first executor failure.
For streaming, measure consumer lag over time rather than at a point (kafka-consumer-groups --describe). Lag that grows linearly means throughput, lag that spikes and recovers means a poison message or a GC pause.
Confirm whether the job failed or was killed. YARN, Kubernetes and Databricks all kill containers that exceed memory limits, and the resulting message looks like a crash rather than an eviction.
Before increasing memory, try increasing partitions. Doubling spark.sql.shuffle.partitions is cheaper and more often correct than doubling executor memory.

Tools worth reaching for

Spark UI
kafka-consumer-groups
Airflow task logs
EXPLAIN / query profile

Authoritative references

Primary documentation for this error, worth reading before applying any fix in production.

spark.apache.org spark.apache.org

Related BigData errors

See all 17 BigData errors →

Browse other categories

Something missing or wrong?

This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.