Monitoring & Observability Errors

Understanding Monitoring errors

Monitoring failures create a dangerous blind spot: dashboards go flat and the natural assumption is that traffic stopped. The usual causes are scrape targets unreachable (network policy, wrong port, or a service with no endpoints), exporter queues full (the backend is rejecting or throttling), and cardinality explosions that cause the time-series database to reject writes. Always alert on the absence of data, not only on bad values.

How to debug Monitoring errors

1. Check the scrape target's status page (/targets in Prometheus). It shows the last scrape error verbatim.
2. Curl the metrics endpoint from inside the cluster or network segment the scraper runs in: reachability from your laptop proves nothing.
3. For OpenTelemetry, enable the collector's own telemetry and watch otelcol_exporter_queue_size and otelcol_exporter_send_failed_spans.
4. Investigate cardinality before increasing memory: a label containing a user ID, request ID or URL path with IDs in it will exhaust any TSDB.
5. Write a dead-man's-switch alert that fires when a known-always-firing metric stops arriving. It is the only alert that catches a broken pipeline.

Tools worth reaching for

• Prometheus /targets
• promtool check config
• otelcol internal metrics
• curl <exporter>/metrics
• Grafana Explore

All 12 Monitoring errors

• CloudWatch Alarm: INSUFFICIENT_DATAA CloudWatch alarm has no data to evaluate against. The metric may not be emitting data, the namespace may be wrong…
• Datadog: Agent not reporting dataThe Datadog Agent is installed but not sending metrics to the Datadog platform. May be caused by API key issues…
• Grafana: Data source connection errorGrafana cannot connect to data source. Wrong URL, authentication failed, or data source unreachable.
• Grafana: Panel plugin not found NewA dashboard references a panel type this Grafana does not have installed. It happens when a dashboard JSON is imported…
• Jaeger: spans droppedThe Jaeger collector is dropping spans because its internal queue is full. The collector cannot process spans fast…
• OpenTelemetry Collector: exporter queue is fullExporter queue/batch processor dropped spans/metrics because downstream endpoint too slow or queue too small.
• OpenTelemetry: connection refused exporting to the collector NewThe SDK could not reach an OTLP endpoint. Almost always the port or protocol is wrong: 4317 is OTLP/gRPC and 4318 is…
• Prometheus: context deadline exceeded (scrape)Prometheus timed out while scraping a target. The target is too slow to respond within the configured scrape_timeout.
• Prometheus: out-of-order sample / duplicate sample for timestamp NewA sample arrived with a timestamp older than the last one stored for that series. Common when two scrapers write the…
• Prometheus: sample limit exceeded NewA scrape returned more series than the job's sample_limit allows, so the whole scrape is dropped and the target goes…
• Prometheus: Target downPrometheus cannot scrape metrics from target endpoint. Service down, network issue, or wrong configuration.
• Sentry: Rate limit exceededThe project has exceeded its Sentry event quota. New error events are being dropped.

Other categories

• AI 35Rate limits, context windows, GPU memory and model-serving failures.
• Ansible 10Unreachable hosts, become passwords, undefined variables and Jinja2 failures.
• API 14Auth headers, payload limits, versioning, idempotency and webhook signatures.
• Apple 10Command line tools, dyld, Homebrew permissions, notarisation and Keychain.
• Auth 11OIDC, SAML, Auth0, Okta, Keycloak, passkeys and MFA failures.
• BigData 11Spark, Kafka, Airflow, Snowflake, Flink and Databricks failures.
• C# 12NuGet restore, null references, EF Core migrations, async deadlocks and Blazor…
• C++ 11Segfaults, linker errors, memory corruption and template deduction failures.
• Caching 10Cache stampedes, stale content, Varnish and CloudFront failures.
• CI/CD 18GitHub Actions, GitLab CI, Jenkins, CircleCI: permissions, runners and…
• Client 21CORS, mixed content, module resolution, memory limits and framework runtime…
• Cloud 25IAM permissions, quotas, service limits and credential failures.
• Dart 10Null safety, pub version solving and build toolchain problems.
• Database 41Connections, deadlocks, constraints, replication and memory limits.
• DNS 10NXDOMAIN, SERVFAIL, timeouts, propagation and delegation problems.
• Docker 27Daemon connectivity, disk space, image pulls, ports and architecture mismatches.
• Elixir 9GenServer timeouts, supervision failures and Mix compilation problems.
• Email 8Delivery failures, relay denial, authentication, SPF, DKIM and DMARC.
• Frontend 23Hydration mismatches, bundler resolution, layout shift and font loading.
• Git 20Merge conflicts, rejected pushes, detached HEAD, LFS and repository corruption.
• Go 19Nil map assignment, concurrent map access, context cancellation and deadlocks.
• GraphQL 13Validation, depth limits, N+1 queries and fragment problems.
• gRPC 10Status codes, deadlines, message limits, TLS and HTTP/2 transport failures.
• HTTP 494xx client errors, 5xx server errors, redirects, headers and protocol problems.
• ICMP 23Destination unreachable, time exceeded, fragmentation needed and redirects.
• Ingress 8404 default backend, missing TLS secrets, IngressClass and path matching.
• Java 19Class loading, dependency resolution, connection pools and JVM version…
• JavaScript 42npm resolution, async pitfalls, hydration, memory limits and runtime type…
• Kubernetes 34CrashLoopBackOff, ImagePullBackOff, OOMKilled, RBAC, scheduling and storage.
• Logging 9Log4j, Logback, Fluentd, Logstash and CloudWatch ingestion problems.
• MessageQueue 14Kafka, RabbitMQ, SQS, NATS and Celery: lag, rebalancing and poison messages.
• Mobile 17Gradle, CocoaPods, Xcode signing, Metro bundler and toolchain mismatches.
• Network 35Refused connections, timeouts, resets, MTU problems and port exhaustion.
• Performance 6GC pauses, thread pool starvation and event loop blocking.
• PHP 11Memory limits, execution timeouts, autoloading, Composer and PDO connections.
• Proxy 17nginx, Envoy, HAProxy, Traefik, Caddy and Cloudflare upstream failures.
• Python 35Imports, virtual environments, encoding, concurrency and dependency conflicts.
• Regex 8Catastrophic backtracking, back references, escaping and engine differences.
• Ruby 11Bundler, migrations, native extensions, encoding and asset compilation.
• Rust 19Borrow checker, ownership moves, trait bounds and lifetime mismatches.
• Scala 6Dependency resolution, binary compatibility and type inference failures.
• Security 25JWT validation, CSRF, OAuth grants, SELinux, SSH host keys and CSP.
• Serverless 11Lambda timeouts, package size limits, VPC networking and cold starts.
• Shell 13Command not found, permissions, quoting, expansion and Makefile syntax.
• Storage 13S3 permissions, NFS mounts, quotas, signed URLs and volume attachment.
• Svelte 10Store subscriptions, load functions and server/client boundaries.
• System 26Disk space, systemd units, file descriptors, OOM killer and scheduled jobs.
• Terraform 18State locks, provider auth, drift, dependency cycles and plan-time unknowns.
• Testing 18Jest, pytest, JUnit, Cypress and Playwright: fixtures, snapshots and timeouts.
• TLS 24Untrusted authorities, expiry, hostname mismatch, chains and cipher negotiation.
• TypeScript 19Assignability, missing declarations, strict null checks and generic constraints.
• Virtualization 8VirtualBox, VMware, Hyper-V, WSL, KVM and hypervisor conflicts.
• Web3 7Gas estimation, nonce management and reverted transactions.
• WebAssembly 7Compile errors, memory bounds and host binding mismatches.
• WebServer 10nginx, Apache, IIS and Caddy: binding, permissions, rewrites and TLS.
• Windows 10Installer failures, missing runtimes, update errors and permission problems.