Monitoring New Added 15 October 2026

Alertmanager: Notify for alerts failed: notify retry canceled after 16 attempts

Alertmanager tried to deliver a notification until the group's retry window ran out, and every attempt failed. The line itself only says it gave up; the reason follows it, usually a webhook returning 4xx, a Slack or Teams URL that was revoked, an SMTP relay rejecting the login, or DNS and proxy problems from inside the cluster. Alerts are firing correctly, they are just not reaching anyone.

Quick fix

Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.

Quick fix

# The cause is in the same log line, after the attempt count
kubectl logs deploy/alertmanager | grep -i 'notify retry canceled' | tail -3

# Count failures per integration
curl -s http://localhost:9093/metrics | grep alertmanager_notifications_failed_total

# Test the receiver URL from inside the Alertmanager pod or host
kubectl exec deploy/alertmanager -- wget -qO- --post-data='{"text":"test"}' https://hooks.slack.com/services/XXX

# Validate the config after fixing it
amtool check-config alertmanager.yml

# Send a test alert end to end
amtool alert add test_alert severity=critical --alertmanager.url=http://localhost:9093

How to diagnose Monitoring errors

Monitoring failures create a dangerous blind spot: dashboards go flat and the natural assumption is that traffic stopped. The usual causes are scrape targets unreachable (network policy, wrong port, or a service with no endpoints), exporter queues full (the backend is rejecting or throttling), and cardinality explosions that cause the time-series database to reject writes. Always alert on the absence of data, not only on bad values.

If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.

Check the scrape target's status page (/targets in Prometheus). It shows the last scrape error verbatim.
Curl the metrics endpoint from inside the cluster or network segment the scraper runs in: reachability from your laptop proves nothing.
For OpenTelemetry, enable the collector's own telemetry and watch otelcol_exporter_queue_size and otelcol_exporter_send_failed_spans.
Investigate cardinality before increasing memory: a label containing a user ID, request ID or URL path with IDs in it will exhaust any TSDB.
Write a dead-man's-switch alert that fires when a known-always-firing metric stops arriving. It is the only alert that catches a broken pipeline.

Tools worth reaching for

Prometheus /targets
promtool check config
otelcol internal metrics
curl <exporter>/metrics
Grafana Explore

Authoritative references

Primary documentation for this error, worth reading before applying any fix in production.

prometheus.io

Related Monitoring errors

See all 18 Monitoring errors →

Browse other categories

Something missing or wrong?

This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.