Monitoring New Added 8 September 2026

Prometheus: sample limit exceeded

A scrape returned more series than the job's sample_limit allows, so the whole scrape is dropped and the target goes down. It is nearly always unbounded cardinality: a label carrying a user id, a URL path or a trace id that creates a new series per request.

Quick fix

Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.

Quick fix

# What is exploding?
curl -s http://target:9090/metrics | wc -l
curl -s http://target:9090/metrics | cut -d'{' -f1 | sort | uniq -c | sort -rn | head

# Fix the instrumentation: labels must have bounded values
# Wrong: path="/users/12345"   Right: route="/users/:id"

# Drop the worst offenders at scrape time
metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'app_request_duration_bucket'
    action: drop

# Raise the limit only once cardinality is understood
sample_limit: 50000

How to diagnose Monitoring errors

Monitoring failures create a dangerous blind spot: dashboards go flat and the natural assumption is that traffic stopped. The usual causes are scrape targets unreachable (network policy, wrong port, or a service with no endpoints), exporter queues full (the backend is rejecting or throttling), and cardinality explosions that cause the time-series database to reject writes. Always alert on the absence of data, not only on bad values.

If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.

Check the scrape target's status page (/targets in Prometheus). It shows the last scrape error verbatim.
Curl the metrics endpoint from inside the cluster or network segment the scraper runs in: reachability from your laptop proves nothing.
For OpenTelemetry, enable the collector's own telemetry and watch otelcol_exporter_queue_size and otelcol_exporter_send_failed_spans.
Investigate cardinality before increasing memory: a label containing a user ID, request ID or URL path with IDs in it will exhaust any TSDB.
Write a dead-man's-switch alert that fires when a known-always-firing metric stops arriving. It is the only alert that catches a broken pipeline.

Tools worth reaching for

Prometheus /targets
promtool check config
otelcol internal metrics
curl <exporter>/metrics
Grafana Explore

Authoritative references

Primary documentation for this error, worth reading before applying any fix in production.

prometheus.io

Related Monitoring errors

See all 12 Monitoring errors →

Browse other categories

Something missing or wrong?

This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.