Celery: WorkerLostError

Quick fix

Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.

# Increase worker memory limit
celery -A proj worker --max-memory-per-child=200000
# Enable task soft time limit
@app.task(soft_time_limit=300, time_limit=600)
def my_task(): ...
# Check system OOM killer
dmesg | grep -i oom
# Use prefork pool with max tasks
--max-tasks-per-child=100

How to diagnose MessageQueue errors

Queue errors are usually about the balance between producers and consumers. Growing lag means consumers cannot keep up. Endless rebalancing usually means consumers are being evicted for exceeding a poll or heartbeat interval, not that the cluster is unhealthy. A poison message, one that always fails processing, will block an ordered partition forever unless there is a dead-letter path, which is why a DLQ is not optional in production.

If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.

1. Measure lag over time, not once: kafka-consumer-groups --describe --group <g>. The trend distinguishes a throughput problem from a stuck consumer.
2. For repeated rebalancing, compare processing time per batch against max.poll.interval.ms. Slow processing looks exactly like a dead consumer to the coordinator.
3. Always configure a dead-letter queue with a bounded retry count. Without one, a single malformed message stops the partition indefinitely.
4. Check message size against the broker limit before assuming a network failure: SQS caps at 256 KB, Kafka at message.max.bytes.
5. Verify consumer ACLs and topic authorisation separately from connectivity; an authorisation failure often surfaces as a metadata error.

Tools worth reaching for

• kafka-consumer-groups
• rabbitmqctl list_queues
• aws sqs get-queue-attributes
• kafkacat / kcat
• flower (Celery)

Authoritative references

Primary documentation for this error, worth reading before applying any fix in production.

📖 docs.celeryq.dev

Related MessageQueue errors

• AWS SQS: Message too large (256KB limit)The SQS message body exceeds the 256KB maximum size limit.
• Kafka: LEADER_NOT_AVAILABLEPartition has no elected leader. Broker down, ISR empty, or topic just created and metadata…
• Kafka: Offset out of rangeConsumer trying to read from offset that doesn't exist. Data retention deleted old messages.
• Kafka: RecordTooLargeExceptionA message exceeded a size limit. Three separate limits must agree: the producer's…
• Kafka: Topic authorization failedThe Kafka client does not have the required ACL permissions to produce to or consume from the…
• Kafka: UNKNOWN_TOPIC_OR_PARTITIONThe client asked for a topic the broker does not know about. Either the topic genuinely does…
• Message Queue: Consumer lag too highMessages accumulating faster than consumers can process. Queue backlog growing.
• Message Queue: Poison message blocking queueMessage causing consumer to crash repeatedly. Message stuck in queue blocking other messages.

See all 14 MessageQueue errors →

Browse other categories

• HTTP 494xx client errors, 5xx server errors, redirects, headers and protocol problems.
• JavaScript 42npm resolution, async pitfalls, hydration, memory limits and runtime type…
• Database 41Connections, deadlocks, constraints, replication and memory limits.
• AI 35Rate limits, context windows, GPU memory and model-serving failures.
• Network 35Refused connections, timeouts, resets, MTU problems and port exhaustion.
• Python 35Imports, virtual environments, encoding, concurrency and dependency conflicts.
• Kubernetes 34CrashLoopBackOff, ImagePullBackOff, OOMKilled, RBAC, scheduling and storage.
• Docker 27Daemon connectivity, disk space, image pulls, ports and architecture mismatches.
• System 26Disk space, systemd units, file descriptors, OOM killer and scheduled jobs.
• Cloud 25IAM permissions, quotas, service limits and credential failures.
• Security 25JWT validation, CSRF, OAuth grants, SELinux, SSH host keys and CSP.
• TLS 24Untrusted authorities, expiry, hostname mismatch, chains and cipher negotiation.

Something missing or wrong?

This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.