First requests after a deploy are slow, then it settles
A freshly started process serves unoptimised code until the JIT has enough samples, with empty connection pools, empty caches and a cold page cache underneath. Rolling the deployment moves the slow requests around rather than removing them, so the graph looks like the load balancer picking bad instances.
Quick fix
Read the commands before running them. Anything that restarts a service, deletes data or changes permissions should be tried on a non-production system first.
# Confirm it is warm up and not one bad instance: the curve should decay
for i in $(seq 1 50); do curl -s -o /dev/null -w '%{time_total}\n' localhost:8080/api/search?q=a; done
# Warm up before the instance takes traffic, and make readiness mean ready
# readinessProbe hitting a path that touches the database and the cache
# httpGet: { path: /ready, port: 8080 }
# periodSeconds: 2
# Startup work in the process itself, before the port is announced
curl -s localhost:8080/api/search?q=warmup >/dev/null
# Open the pool at boot instead of on the first request
# HikariCP minimumIdle, pgbouncer min_pool_size, HTTP clients keep alive
# JVM: give the compiler room so the profile is not thrown away
java -XX:ReservedCodeCacheSize=256m -jar app.jar
# .NET: dotnet publish -p:PublishReadyToRun=true
# Then compare the first and the hundredth request, not the average
curl -s -o /dev/null -w '%{time_total}\n' localhost:8080/api/search?q=a
How to diagnose Performance errors
Performance failures rarely produce an error message. They produce timeouts elsewhere. The three classic causes are stop-the-world garbage collection, thread pool starvation (all workers blocked on I/O so new requests queue), and event loop blocking in single-threaded runtimes. All three look identical from outside: latency climbs, then upstream timeouts fire. Distinguishing them requires looking inside the process.
If the quick fix above does not resolve it, work through these steps. They apply to this whole class of error, not just to this one message, which is usually what saves the time.
- Measure the p99, not the mean. Averages hide exactly the pauses that cause timeouts.
- For the JVM, enable GC logging (
-Xlog:gc*) and correlate pause durations with latency spikes before tuning anything. - For thread pools, log active versus queued task counts. A queue that grows while CPU is idle is starvation, and the fix is asynchronous I/O, not more threads.
- In Node.js, measure event loop lag directly (
perf_hooks.monitorEventLoopDelay). Any synchronous work over a few milliseconds per request will show up here. - Profile before optimising.
pprof,async-profiler,py-spyand Chrome's profiler all point at the real hot path, which is rarely where intuition suggests.
Tools worth reaching for
async-profilergo tool pprofpy-spyperf_hooks.monitorEventLoopDelay-Xlog:gc*
Authoritative references
Primary documentation for this error, worth reading before applying any fix in production.
Related Performance errors
- Connections hang or reset with ListenOverflows increasingThe kernel finished the handshake but the application was too slow to call accept(), so the…
- Container CPU throttling: nr_throttled rising in cpu.statA CPU limit is enforced as a quota per 100ms period, so a process that spends its quota early…
- CPU0: Core temperature above threshold, cpu clock throttledThe CPU hit its thermal limit and the hardware cut its clock speed to cool down. Everything…
- Go: high tail latency in a container, GOMAXPROCS above the CPU limitBefore Go 1.25, GOMAXPROCS defaulted to the number of CPUs on the node and ignored the…
- High iowait: load average climbing while the CPU sits idleLinux counts tasks in uninterruptible sleep towards the load average, so processes blocked on…
- INFO: task kworker blocked for more than 120 secondsThe hung task detector found a process stuck in uninterruptible sleep (D state) for two…
- java.lang.OutOfMemoryError: GC overhead limit exceededThe JVM spent over 98 percent of recent time collecting and recovered less than 2 percent of…
- JVM: pause with no GC in the log (long time to safepoint)A safepoint stops every application thread, and the pause includes however long the slowest…
Browse other categories
- HTTP 494xx client errors, 5xx server errors, redirects, headers and protocol problems.
- JavaScript 42npm resolution, async pitfalls, hydration, memory limits and runtime type…
- Database 41Connections, deadlocks, constraints, replication and memory limits.
- AI 35Rate limits, context windows, GPU memory and model-serving failures.
- Network 35Refused connections, timeouts, resets, MTU problems and port exhaustion.
- Python 35Imports, virtual environments, encoding, concurrency and dependency conflicts.
- Kubernetes 34CrashLoopBackOff, ImagePullBackOff, OOMKilled, RBAC, scheduling and storage.
- Docker 27Daemon connectivity, disk space, image pulls, ports and architecture mismatches.
- System 26Disk space, systemd units, file descriptors, OOM killer and scheduled jobs.
- Cloud 25IAM permissions, quotas, service limits and credential failures.
- Security 25JWT validation, CSRF, OAuth grants, SELinux, SSH host keys and CSP.
- TLS 24Untrusted authorities, expiry, hostname mismatch, chains and cipher negotiation.
Something missing or wrong?
This entry is maintained by hand. If the fix is out of date, incomplete, or you have a better one, email a correction and it will be reviewed.