Monitoring New Added 2 October 2026

Prometheus: many-to-many matching not allowed: matching labels must be unique

You combined two vectors with a binary operator and on the labels Prometheus matched on, both sides had more than one series. Each result has to come from exactly one pair. The query may have worked for weeks and then broken when a second instance, a restarted pod or a new label value added another series with the same labels.

Quick fix

Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.

Quick fix

# Find which side has duplicates for the match labels
count by (instance) (node_uname_info) > 1
count by (instance) (up{job="node"}) > 1

# Match on fewer labels, and collapse the 'one' side so it is unique
rate(http_requests_total[5m])
  * on (instance) group_left (version)
  max by (instance, version) (app_build_info)

# Or drop the label that differs between duplicates
sum without (pod) (metric_a) / sum without (pod) (metric_b)

How to diagnose Monitoring errors

Monitoring failures create a dangerous blind spot: dashboards go flat and the natural assumption is that traffic stopped. The usual causes are scrape targets unreachable (network policy, wrong port, or a service with no endpoints), exporter queues full (the backend is rejecting or throttling), and cardinality explosions that cause the time-series database to reject writes. Always alert on the absence of data, not only on bad values.

If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.

Check the scrape target's status page (/targets in Prometheus). It shows the last scrape error verbatim.
Curl the metrics endpoint from inside the cluster or network segment the scraper runs in: reachability from your laptop proves nothing.
For OpenTelemetry, enable the collector's own telemetry and watch otelcol_exporter_queue_size and otelcol_exporter_send_failed_spans.
Investigate cardinality before increasing memory: a label containing a user ID, request ID or URL path with IDs in it will exhaust any TSDB.
Write a dead-man's-switch alert that fires when a known-always-firing metric stops arriving. It is the only alert that catches a broken pipeline.

Tools worth reaching for

Prometheus /targets
promtool check config
otelcol internal metrics
curl <exporter>/metrics
Grafana Explore

Authoritative references

Primary documentation for this error, worth reading before applying any fix in production.

prometheus.io

Related Monitoring errors

See all 18 Monitoring errors →

Browse other categories

Something missing or wrong?

This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.