Linux & System Errors
Disk space, systemd units, file descriptors, OOM killer and scheduled jobs.
Understanding System errors
System-level errors are usually a resource limit being hit: disk, inodes, file descriptors, memory or process count. Two traps recur. "No space left on device" can mean inodes are exhausted while df -h shows free space. And a process that vanished without an error in its own log was almost certainly killed by the OOM killer, which logs to the kernel ring buffer rather than to the application.
How to debug System errors
- Check both space and inodes:
df -handdf -i. Millions of tiny files exhaust inodes long before bytes. - For processes that disappear, search the kernel log:
dmesg -T | grep -i -E 'oom|killed process'orjournalctl -k. - For systemd units,
systemctl status unitthenjournalctl -u unit -n 200 --no-pager. The status output truncates the useful part. - For "too many open files", check both the limit and the usage:
ulimit -n,cat /proc/<pid>/limits, andls /proc/<pid>/fd | wc -l. systemd services needLimitNOFILE, not a shell ulimit. - For cron jobs that work manually but not scheduled, remember cron has a minimal environment and no PATH. Use absolute paths and log stdout and stderr to a file.
Tools worth reaching for
df -h / df -idmesg -Tjournalctl -ulsof / sssystemd-analyze blamencdu
All 26 System errors
- apt: Could not get lock /var/lib/dpkg/lock-frontend NewAnother process holds the package manager lock. On a freshly booted Ubuntu machine it is nearly always…
- bash: fork: retry: Resource temporarily unavailable NewThe kernel refused to create another process because a limit was reached: the per user process count, the cgroup's…
- Cron: Job not runningCron job not executing. Syntax error, permission issues, or cron daemon not running.
- ENOSPC: System limit for number of file watchers reached NewA dev server or file watcher exhausted the per-user inotify watch limit. Despite the ENOSPC name there is no disk…
- Go: import cycle not allowedCircular dependency between packages. Package A imports Package B, and Package B imports Package A.
- Go: panic: runtime error: invalid memory address or nil pointer dereferenceAttempting to dereference a nil pointer. Often happens when initialising a pointer but not assigning a value, or…
- GPG: BAD signature / cannot verify signatureThe signature does not validate with the public key you have. The file may be modified, truncated, signed with another…
- Linux: No space left on deviceFile system is full. Cannot write files or create directories.
- Linux: Read-only file system NewThe filesystem was mounted read-only, either intentionally or because the kernel remounted it after detecting I/O…
- Linux: Too many levels of symbolic links (ELOOP) NewPath resolution followed more than forty symlinks and gave up, which means a loop. The classic case is a link that…
- macOS: app is damaged and can't be opened / Gatekeeper blockedmacOS quarantined a downloaded app or binary because it is unsigned, unnotarized, or not trusted yet.
- PowerShell: .ps1 cannot be loaded because running scripts is disabledWindows default security policy blocks script execution to prevent malware.
- Rust: borrow of moved valueAttempting to use a value after it has been moved (ownership transferred). Rust ownership rules prevent using a…
- Rust: lifetime mismatchReferences in function have different lifetimes but are required to be related. Rust borrow checker cannot guarantee…
- ssh: Could not open a connection to your authentication agentssh-add cannot find a running ssh-agent because SSH_AUTH_SOCK is unset or points to a stale socket.
- sudo: unable to resolve host NewThe hostname in /etc/hostname has no entry in /etc/hosts, so sudo's lookup fails. It is a warning rather than a…
- System: OOM killer terminated processLinux Out-Of-Memory killer terminated process consuming too much memory.
- System: Segmentation fault (core dumped)Program attempted to access memory it wasn't allowed to. Invalid memory access, buffer overflow, or stack overflow.
- System: Too many open filesProcess exceeded file descriptor limit. Need to increase ulimit or close unused files.
- System: Zombie process accumulationParent process not reaping child processes. Zombie processes accumulating in system.
- systemd: Failed to start (unit is masked) NewThe unit is symlinked to /dev/null, which makes it impossible to start by any means including as a dependency. Masking…
- systemd: Start operation timed outService took longer than configured timeout to start. Increase timeout or fix slow startup.
- systemd: start request repeated too quickly, refusing to start NewThe unit crashed and restarted more times than StartLimitBurst allows within StartLimitIntervalSec, so systemd stopped…
- systemd: Unit failed to startsystemd service failed to start. Configuration error, dependency issues, or permission problems.
- Windows: MSI installer error 1603A generic fatal MSI installation failure. The real error is usually near 'Return value 3' in verbose logs.
- WSL2: Error 0x80370102 / vmcompute not runningWSL2 cannot start its VM because virtualization features are disabled, hypervisorlaunchtype is off, or vmcompute is…
Other categories
- AI 35Rate limits, context windows, GPU memory and model-serving failures.
- Ansible 10Unreachable hosts, become passwords, undefined variables and Jinja2 failures.
- API 14Auth headers, payload limits, versioning, idempotency and webhook signatures.
- Apple 10Command line tools, dyld, Homebrew permissions, notarisation and Keychain.
- Auth 11OIDC, SAML, Auth0, Okta, Keycloak, passkeys and MFA failures.
- BigData 11Spark, Kafka, Airflow, Snowflake, Flink and Databricks failures.
- C# 12NuGet restore, null references, EF Core migrations, async deadlocks and Blazor…
- C++ 11Segfaults, linker errors, memory corruption and template deduction failures.
- Caching 10Cache stampedes, stale content, Varnish and CloudFront failures.
- CI/CD 18GitHub Actions, GitLab CI, Jenkins, CircleCI: permissions, runners and…
- Client 21CORS, mixed content, module resolution, memory limits and framework runtime…
- Cloud 25IAM permissions, quotas, service limits and credential failures.
- Dart 10Null safety, pub version solving and build toolchain problems.
- Database 41Connections, deadlocks, constraints, replication and memory limits.
- DNS 10NXDOMAIN, SERVFAIL, timeouts, propagation and delegation problems.
- Docker 27Daemon connectivity, disk space, image pulls, ports and architecture mismatches.
- Elixir 9GenServer timeouts, supervision failures and Mix compilation problems.
- Email 8Delivery failures, relay denial, authentication, SPF, DKIM and DMARC.
- Frontend 23Hydration mismatches, bundler resolution, layout shift and font loading.
- Git 20Merge conflicts, rejected pushes, detached HEAD, LFS and repository corruption.
- Go 19Nil map assignment, concurrent map access, context cancellation and deadlocks.
- GraphQL 13Validation, depth limits, N+1 queries and fragment problems.
- gRPC 10Status codes, deadlines, message limits, TLS and HTTP/2 transport failures.
- HTTP 494xx client errors, 5xx server errors, redirects, headers and protocol problems.
- ICMP 23Destination unreachable, time exceeded, fragmentation needed and redirects.
- Ingress 8404 default backend, missing TLS secrets, IngressClass and path matching.
- Java 19Class loading, dependency resolution, connection pools and JVM version…
- JavaScript 42npm resolution, async pitfalls, hydration, memory limits and runtime type…
- Kubernetes 34CrashLoopBackOff, ImagePullBackOff, OOMKilled, RBAC, scheduling and storage.
- Logging 9Log4j, Logback, Fluentd, Logstash and CloudWatch ingestion problems.
- MessageQueue 14Kafka, RabbitMQ, SQS, NATS and Celery: lag, rebalancing and poison messages.
- Mobile 17Gradle, CocoaPods, Xcode signing, Metro bundler and toolchain mismatches.
- Monitoring 12Prometheus scrapes, Grafana data sources, OpenTelemetry exporters and agent…
- Network 35Refused connections, timeouts, resets, MTU problems and port exhaustion.
- Performance 6GC pauses, thread pool starvation and event loop blocking.
- PHP 11Memory limits, execution timeouts, autoloading, Composer and PDO connections.
- Proxy 17nginx, Envoy, HAProxy, Traefik, Caddy and Cloudflare upstream failures.
- Python 35Imports, virtual environments, encoding, concurrency and dependency conflicts.
- Regex 8Catastrophic backtracking, back references, escaping and engine differences.
- Ruby 11Bundler, migrations, native extensions, encoding and asset compilation.
- Rust 19Borrow checker, ownership moves, trait bounds and lifetime mismatches.
- Scala 6Dependency resolution, binary compatibility and type inference failures.
- Security 25JWT validation, CSRF, OAuth grants, SELinux, SSH host keys and CSP.
- Serverless 11Lambda timeouts, package size limits, VPC networking and cold starts.
- Shell 13Command not found, permissions, quoting, expansion and Makefile syntax.
- Storage 13S3 permissions, NFS mounts, quotas, signed URLs and volume attachment.
- Svelte 10Store subscriptions, load functions and server/client boundaries.
- Terraform 18State locks, provider auth, drift, dependency cycles and plan-time unknowns.
- Testing 18Jest, pytest, JUnit, Cypress and Playwright: fixtures, snapshots and timeouts.
- TLS 24Untrusted authorities, expiry, hostname mismatch, chains and cipher negotiation.
- TypeScript 19Assignability, missing declarations, strict null checks and generic constraints.
- Virtualization 8VirtualBox, VMware, Hyper-V, WSL, KVM and hypervisor conflicts.
- Web3 7Gas estimation, nonce management and reverted transactions.
- WebAssembly 7Compile errors, memory bounds and host binding mismatches.
- WebServer 10nginx, Apache, IIS and Caddy: binding, permissions, rewrites and TLS.
- Windows 10Installer failures, missing runtimes, update errors and permission problems.