System: Zombie process accumulation

Quick fix

Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.

# Find zombie processes
ps aux | grep 'Z'
# Kill parent process
kill -9 parent_pid
# In code: properly wait for children
import subprocess
process.wait()  # Python
wait()  // C

How to diagnose System errors

System-level errors are usually a resource limit being hit: disk, inodes, file descriptors, memory or process count. Two traps recur. "No space left on device" can mean inodes are exhausted while df -h shows free space. And a process that vanished without an error in its own log was almost certainly killed by the OOM killer, which logs to the kernel ring buffer rather than to the application.

If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.

1. Check both space and inodes: df -h and df -i. Millions of tiny files exhaust inodes long before bytes.
2. For processes that disappear, search the kernel log: dmesg -T | grep -i -E 'oom|killed process' or journalctl -k.
3. For systemd units, systemctl status unit then journalctl -u unit -n 200 --no-pager. The status output truncates the useful part.
4. For "too many open files", check both the limit and the usage: ulimit -n, cat /proc/<pid>/limits, and ls /proc/<pid>/fd | wc -l. systemd services need LimitNOFILE, not a shell ulimit.
5. For cron jobs that work manually but not scheduled, remember cron has a minimal environment and no PATH. Use absolute paths and log stdout and stderr to a file.

Tools worth reaching for

• df -h / df -i
• dmesg -T
• journalctl -u
• lsof / ss
• systemd-analyze blame
• ncdu

Authoritative references

Primary documentation for this error, worth reading before applying any fix in production.

📖 en.wikipedia.org

Related System errors

• apt: Could not get lock /var/lib/dpkg/lock-frontendAnother process holds the package manager lock. On a freshly booted Ubuntu machine it is…
• bash: fork: retry: Resource temporarily unavailableThe kernel refused to create another process because a limit was reached: the per user…
• Cron: Job not runningCron job not executing. Syntax error, permission issues, or cron daemon not running.
• ENOSPC: System limit for number of file watchers reachedA dev server or file watcher exhausted the per-user inotify watch limit. Despite the ENOSPC…
• Go: import cycle not allowedCircular dependency between packages. Package A imports Package B, and Package B imports…
• Go: panic: runtime error: invalid memory address or nil pointer dereferenceAttempting to dereference a nil pointer. Often happens when initialising a pointer but not…
• GPG: BAD signature / cannot verify signatureThe signature does not validate with the public key you have. The file may be modified…
• Linux: No space left on deviceFile system is full. Cannot write files or create directories.

See all 26 System errors →

Browse other categories

• HTTP 494xx client errors, 5xx server errors, redirects, headers and protocol problems.
• JavaScript 42npm resolution, async pitfalls, hydration, memory limits and runtime type…
• Database 41Connections, deadlocks, constraints, replication and memory limits.
• AI 35Rate limits, context windows, GPU memory and model-serving failures.
• Network 35Refused connections, timeouts, resets, MTU problems and port exhaustion.
• Python 35Imports, virtual environments, encoding, concurrency and dependency conflicts.
• Kubernetes 34CrashLoopBackOff, ImagePullBackOff, OOMKilled, RBAC, scheduling and storage.
• Docker 27Daemon connectivity, disk space, image pulls, ports and architecture mismatches.
• Cloud 25IAM permissions, quotas, service limits and credential failures.
• Security 25JWT validation, CSRF, OAuth grants, SELinux, SSH host keys and CSP.
• TLS 24Untrusted authorities, expiry, hostname mismatch, chains and cipher negotiation.
• Frontend 23Hydration mismatches, bundler resolution, layout shift and font loading.

Something missing or wrong?

This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.