Airflow: Broken DAG: [/opt/airflow/dags/etl.py] No module named 'x'
The DAG processor could not import the file, so the DAG vanishes from the UI and a red banner appears instead. The module is missing from the environment the scheduler or DAG processor runs in, which is often not the one you tested in: a package installed in the webserver image only, or a helper module outside the dags folder and so not on sys.path.
Quick fix
Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.
# Reproduce the import exactly as Airflow does
airflow dags list-import-errors
python /opt/airflow/dags/etl.py
# Run it inside the scheduler or dag-processor container, not your laptop
docker compose exec airflow-scheduler python -c 'import x'
# Missing package: add it to the image every component uses
# requirements.txt, then rebuild
pip install apache-airflow-providers-x==<version>
# Local helper module: put it under dags/ or plugins/ so it is on sys.path
# dags/
# common/__init__.py
# common/helpers.py
# etl.py -> from common.helpers import loadHow to diagnose BigData errors
Distributed data errors are rarely about the line of code that threw them. An executor OOM, a failed checkpoint or a stuck consumer group are usually symptoms of data skew, insufficient partitioning, or a downstream system applying backpressure. The diagnostic instinct that pays off here is to look at the distribution of work across tasks before looking at the exception.
If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.
- Open the Spark UI (or Flink dashboard) and sort tasks by duration and shuffle read size. If one task is an order of magnitude larger than the median, you have skew, not a memory shortage.
- Check the driver log and at least one executor log. The exception the driver reports is often a downstream consequence of the first executor failure.
- For streaming, measure consumer lag over time rather than at a point (
kafka-consumer-groups --describe). Lag that grows linearly means throughput, lag that spikes and recovers means a poison message or a GC pause. - Confirm whether the job failed or was killed. YARN, Kubernetes and Databricks all kill containers that exceed memory limits, and the resulting message looks like a crash rather than an eviction.
- Before increasing memory, try increasing partitions. Doubling
spark.sql.shuffle.partitionsis cheaper and more often correct than doubling executor memory.
Tools worth reaching for
Spark UIkafka-consumer-groupsAirflow task logsEXPLAIN / query profile
Authoritative references
Primary documentation for this error, worth reading before applying any fix in production.
Related BigData errors
- Airflow: DAG.__init__() got an unexpected keyword argument 'schedule_interval'Airflow 3 removed the schedule_interval and timetable arguments that had been deprecated…
- Airflow: Scheduler not runningAirflow scheduler process is down or tasks are stuck in queued state.
- Databricks: Cluster terminated unexpectedlyA Databricks cluster was terminated due to spot instance reclamation, out of memory, or cloud…
- dbt: Compilation Error (model depends on a node that was not found)A ref() or source() points at something dbt cannot resolve: a misspelled model name, a source…
- Flink: Checkpoint failedApache Flink failed to complete a checkpoint within the timeout. Usually caused by…
- Hadoop: NameNode is in safe modeHDFS NameNode is in safe mode, making the filesystem read-only. This happens at startup until…
- Hive: MetaStore connection failedHive cannot connect to its MetaStore service, which stores table schemas and partition info…
- Kafka: Consumer group rebalancingKafka consumer group is stuck in a rebalance loop. Consumers are frequently joining and…
Browse other categories
- HTTP 494xx client errors, 5xx server errors, redirects, headers and protocol problems.
- JavaScript 42npm resolution, async pitfalls, hydration, memory limits and runtime type…
- Database 41Connections, deadlocks, constraints, replication and memory limits.
- AI 35Rate limits, context windows, GPU memory and model-serving failures.
- Network 35Refused connections, timeouts, resets, MTU problems and port exhaustion.
- Python 35Imports, virtual environments, encoding, concurrency and dependency conflicts.
- Kubernetes 34CrashLoopBackOff, ImagePullBackOff, OOMKilled, RBAC, scheduling and storage.
- Docker 27Daemon connectivity, disk space, image pulls, ports and architecture mismatches.
- System 26Disk space, systemd units, file descriptors, OOM killer and scheduled jobs.
- Cloud 25IAM permissions, quotas, service limits and credential failures.
- Security 25JWT validation, CSRF, OAuth grants, SELinux, SSH host keys and CSP.
- TLS 24Untrusted authorities, expiry, hostname mismatch, chains and cipher negotiation.
Something missing or wrong?
This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.