Prometheus: Target down

Quick fix

Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.

# Check target status
curl http://prometheus:9090/targets
# Test endpoint
curl http://target:9090/metrics
# Check Prometheus config
prometheus --config.file=prometheus.yml --dry-run
# Reload config
kill -HUP prometheus-pid

How to diagnose Monitoring errors

Monitoring failures create a dangerous blind spot: dashboards go flat and the natural assumption is that traffic stopped. The usual causes are scrape targets unreachable (network policy, wrong port, or a service with no endpoints), exporter queues full (the backend is rejecting or throttling), and cardinality explosions that cause the time-series database to reject writes. Always alert on the absence of data, not only on bad values.

If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.

1. Check the scrape target's status page (/targets in Prometheus). It shows the last scrape error verbatim.
2. Curl the metrics endpoint from inside the cluster or network segment the scraper runs in: reachability from your laptop proves nothing.
3. For OpenTelemetry, enable the collector's own telemetry and watch otelcol_exporter_queue_size and otelcol_exporter_send_failed_spans.
4. Investigate cardinality before increasing memory: a label containing a user ID, request ID or URL path with IDs in it will exhaust any TSDB.
5. Write a dead-man's-switch alert that fires when a known-always-firing metric stops arriving. It is the only alert that catches a broken pipeline.

Tools worth reaching for

• Prometheus /targets
• promtool check config
• otelcol internal metrics
• curl <exporter>/metrics
• Grafana Explore

Authoritative references

Primary documentation for this error, worth reading before applying any fix in production.

📖 prometheus.io

Related Monitoring errors

• CloudWatch Alarm: INSUFFICIENT_DATAA CloudWatch alarm has no data to evaluate against. The metric may not be emitting data, the…
• Datadog: Agent not reporting dataThe Datadog Agent is installed but not sending metrics to the Datadog platform. May be caused…
• Grafana: Data source connection errorGrafana cannot connect to data source. Wrong URL, authentication failed, or data source…
• Grafana: Panel plugin not foundA dashboard references a panel type this Grafana does not have installed. It happens when a…
• Jaeger: spans droppedThe Jaeger collector is dropping spans because its internal queue is full. The collector…
• OpenTelemetry Collector: exporter queue is fullExporter queue/batch processor dropped spans/metrics because downstream endpoint too slow or…
• OpenTelemetry: connection refused exporting to the collectorThe SDK could not reach an OTLP endpoint. Almost always the port or protocol is wrong: 4317…
• Prometheus: context deadline exceeded (scrape)Prometheus timed out while scraping a target. The target is too slow to respond within the…

See all 12 Monitoring errors →

Browse other categories

• HTTP 494xx client errors, 5xx server errors, redirects, headers and protocol problems.
• JavaScript 42npm resolution, async pitfalls, hydration, memory limits and runtime type…
• Database 41Connections, deadlocks, constraints, replication and memory limits.
• AI 35Rate limits, context windows, GPU memory and model-serving failures.
• Network 35Refused connections, timeouts, resets, MTU problems and port exhaustion.
• Python 35Imports, virtual environments, encoding, concurrency and dependency conflicts.
• Kubernetes 34CrashLoopBackOff, ImagePullBackOff, OOMKilled, RBAC, scheduling and storage.
• Docker 27Daemon connectivity, disk space, image pulls, ports and architecture mismatches.
• System 26Disk space, systemd units, file descriptors, OOM killer and scheduled jobs.
• Cloud 25IAM permissions, quotas, service limits and credential failures.
• Security 25JWT validation, CSRF, OAuth grants, SELinux, SSH host keys and CSP.
• TLS 24Untrusted authorities, expiry, hostname mismatch, chains and cipher negotiation.

Something missing or wrong?

This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.