$ guides / nginx ▌

NGINX · OPERATIONS PLAYBOOK

Running NGINX in production, without the 502 at 3 a.m.

Master and workers, the event loop, connection slots, upstreams. The mental model of how the server actually processes traffic, where it tends to break, what to monitor as your operation matures, and the runbooks for the incidents you'll see.

> Start with the monitoring checklist → # Jump to the full guide list

NGINX almost never breaks on its own. It breaks at the seams — between the kernel and the worker, between the worker and a slow backend, between a buffer and the disk.

The defaults work. Until a slow upstream makes connections pile up and every worker_connections slot is consumed. Until the process hits the default 1024 file-descriptor limit and starts logging too many open files. Until a backend stops responding and nginx returns 502 for traffic that has nothing wrong with it. Until a large response overflows proxy_buffers and silently spills to a temp file while latency triples. Until the kernel listen queue overflows and connections are dropped before nginx ever sees them — with zero evidence in any nginx log.

These guides are written for engineers who already run NGINX, not for people learning what a reverse proxy is. The goal is the mental model of how workers actually process traffic, the failure patterns that keep recurring, the monitoring story that catches problems before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How NGINX actually runs in production

NGINX is not one thing doing the work. It is a master process that binds ports and supervises, a handful of single-threaded worker event loops that do all the traffic, a per-connection state machine, and a strict contract with the kernel below and the backends above. Most production failures live between these layers, not inside any one of them.

clients / browsers

Whatever opens connections: browsers, mobile apps, API clients, other proxies, load-balancer health checks. Slow clients and aggressive client timeouts shape what nginx sees as latency and 499s.

CLIENT

kernel listen queue

The TCP accept queue, sized by <code>net.core.somaxconn</code> and the <code>listen</code> <code>backlog</code> (effective = the lower of the two). When it overflows, connections are dropped here — before any worker sees them, invisible to nginx logs.

KERNEL

master process

Reads config, binds listening sockets, spawns workers, and handles signal-based lifecycle (reload, reopen, binary upgrade). It never processes traffic. A failed reload does not stop it — the old config keeps serving.

MASTER

worker event loop

One single-threaded <code>epoll</code> loop per worker. Each holds <code>worker_connections</code> slots (default 512, not 1024). A worker never blocks — it registers interest in the next event and moves on. CPU-bound work (TLS, gzip, regex) stalls everything that worker holds.

WORKER

connection state machine

Every connection moves through reading headers, reading body, processing, writing response, keepalive idle. Per-connection buffers (<code>client_body_buffer_size</code>, <code>proxy_buffers</code>) live here; overflow spills to disk temp files.

STATE

upstream pool

Per-worker keepalive connections to backends. Passive health checking (<code>max_fails</code>, <code>fail_timeout</code>) marks servers down on real-request failures. Every proxied request uses two connections — client side and upstream side.

UPSTREAM

backends / origin

App servers, FastCGI/uWSGI pools, other proxies, object storage. When they slow down or close early, nginx is the messenger — 502, 504, <code>no live upstreams</code> — not the cause.

BACKEND

disk + logs

Request/response temp files, the on-disk proxy cache, and synchronous access/error logs. Under an error storm, log I/O pressures the event loop; a full log disk silences diagnostics exactly when you need them.

DISK

Why this matters: a request can be slow because of a slow client, a slow backend, a temp-file spill to disk, a TLS handshake burst, a regex backtrack, a full listen queue, or a keepalive pool that never reuses connections. The symptom is the same — high $request_time — but each layer has a different signal and a different fix. The classic mistake is reading $request_time as backend latency when it also includes client time.

The failures you'll actually see

Most NGINX incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The connection exhaustion cliff

Every worker_connections slot is in use. nginx accepts connections from the kernel but has no structure to process them, so it drops them. There is no graceful degradation — it is a cliff. Underneath it is usually a slow backend piling up connections, keepalive idle accumulation, or the default 512 slots being far too low for a reverse-proxy workload (where each request needs two).

accepts > handled gap growing in stub_status
active connections near worker_connections × worker_processes
worker_connections are not enough in the error log
request rate dropping while traffic still arrives

Investigate →

IMMINENT

The backend cascade

One upstream slows down. Workers hold connections waiting for it; slots fill; the remaining healthy backends take all the traffic and overload too. nginx looks down even though nginx itself is fine — its CPU is normal and $upstream_response_time is the entire story. If every backend fails, every request becomes a 502 or 504.

$upstream_response_time climbing while nginx CPU is flat
502 / 504 rate rising
no live upstreams while connecting to upstream
active connections climbing with falling throughput

Investigate →

CRITICAL

File descriptor exhaustion

nginx hits the file-descriptor ceiling — the lower of worker_rlimit_nofile and the system ulimit, often a dangerously low 1024. It can no longer accept connections, open log files, or connect to upstreams. Failures are immediate and total; if FDs are truly gone, nginx cannot even log the error. Each proxied request burns two FDs, and idle keepalive connections hold them.

too many open files in the error log
FD usage above 95% of the limit
connections refused despite low CPU
cannot open log file or connect to upstream

Investigate →

WATCHFUL

Buffer spill to disk

A large request or response body exceeds the configured buffers and nginx silently writes it to a temp file. Disk I/O spikes; latency for large transfers explodes. There is no error-log entry — the only fingerprint is a gap where $request_time is far larger than $upstream_response_time. The fix is buffer sizing or faster temp storage, not chasing a phantom backend problem.

request body buffered to a temporary file warnings
$request_time ≫ $upstream_response_time on large transfers
disk I/O correlated with response size
temp files appearing in proxy_temp_path / client_body_temp_path

Investigate →

ACTIVE

SSL termination overload

Workers spend all their CPU on TLS handshakes. It shows up with high connection churn (short-lived connections, no keepalive), a low session-cache hit rate, and throughput that is low relative to connection rate. The bottleneck is CPU, not connections. The fix is session resumption, TLS 1.3, HTTP/2, or moving termination to a dedicated edge — not more workers.

worker CPU near 100% with normal disk and network
SSL session cache hit rate low
connection rate high, requests/sec low
latency climbing in step with new-connection rate

Investigate →

IMMINENT

Silent kernel listen-queue drops

Workers can't accept fast enough, or the backlog is too small, and the kernel drops connections before nginx ever sees them. stub_status shows nothing. The error log shows nothing. The only symptom is client-side connection timeouts, usually blamed on the network. The tell is the TcpExtListenOverflows counter climbing — a signal that lives in the kernel, not in nginx.

TcpExtListenOverflows increasing
ss -tlnp Recv-Q approaching Send-Q on :80/:443
client connection timeouts with clean nginx logs
SYN_RECV socket count growing

Investigate →

NGINX monitoring maturity levels

NGINX observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your edge matters. Most production deployments should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is nginx still serving? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for internal tools and low-stakes sites.

Master process alive + worker count Is the master up and are workers at the configured count?
Listening on expected ports Is :80/:443 actually bound and accepting?
HTTP 5xx rate Are server-side errors spiking?
Active connections (stub_status) Any connections being handled at all?
Error log [emerg]/[alert]/[crit] Did something catastrophic just log?
SSL certificate expiry countdown Is a cert about to take the site down?

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production edges should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: connection pressure, backend latency, error breakdown, client abandons.

Requests per second Is throughput tracking expectations?
Reading / Writing / Waiting breakdown Slow clients, slow upstreams, or healthy keepalive?
Dropped connections (accepts − handled) Is nginx dropping what the kernel handed it?
$request_time percentiles (P50/P95/P99) How slow is the slow tail?
$upstream_response_time percentiles Is the latency in nginx or in the backend?
Status code distribution (4xx / 5xx) Which errors, and how many?
Client abandons (499 rate) Are users bailing before the response?
File descriptor utilisation per worker How close to the FD ceiling?
Worker CPU and RSS Is a worker pinned or leaking?
Error log rate by severity Is the error rate trending up?

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone. A single degrading upstream that aggregate metrics hide. The listen queue starting to overflow. The keepalive pool that stopped reusing connections. The cache hit rate drifting down. None of these page you on day one; they become incidents on day thirty.

Per-upstream metrics ($upstream_addr) Is one backend of five degrading?
$upstream_connect_time / header_time split Is it connect, processing, or transfer?
TcpExtListenOverflows rate Is the kernel dropping connections silently?
Connection slot utilisation ratio How much headroom before the cliff?
Upstream keepalive reuse (connect_time 0) Is the pool actually reusing connections?
Cache hit rate + status distribution HIT / MISS / STALE / EXPIRED breakdown.
SSL session cache hit rate Are handshakes being amortised?
Reload frequency and success/failure Are reloads applying — or silently failing?
Rate-limit rejection rate Are limits firing on real users?
Log-partition disk free Will an error storm fill the disk?

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Per-worker load imbalance, conntrack table pressure, temp-file creation rate, ephemeral-port exhaustion to upstreams, error-masking between $status and $upstream_status. Most teams never need every signal here. Add the ones your incident history demands.

Per-worker CPU / connection imbalance Is one worker doing all the work?
Kernel conntrack table utilisation Is nf_conntrack filling under load?
Temp-file creation rate Are buffers overflowing to disk?
Ephemeral port / TIME_WAIT per upstream Is connection churn exhausting ports?
$upstream_status vs $status discrepancy Are retries masking real backend errors?
Old worker accumulation after reload Are old workers stuck on long-lived connections?
Rate-limit zone effective capacity Have unique keys outgrown the zone?
TLS version / cipher distribution shifts Are legacy clients or attacks shifting the mix?

Operating mistakes worth avoiding

The traps NGINX teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Assuming worker_connections defaults to 1024

The actual default is <code>512</code>. Blog posts and tutorials perpetuate the 1024 myth. Worse, in reverse-proxy mode each request uses two connections (client + upstream), so effective capacity is half what the number suggests. Verify with <code>nginx -T | grep worker_connections</code> and size for the 2× multiplier.

Never monitoring accepts − handled

The dropped-connection counter in stub_status is the canary for connection exhaustion — it grows before active connections hit the ceiling. It is zero-cost, always available, and almost universally ignored. Track the rate of the gap, not the absolute value.

Reading $request_time as backend latency

The single most common nginx misdiagnosis. <code>$request_time</code> includes client send/receive time, so a 50 ms backend response delivered to a slow mobile client logs as 5 s. Always log and compare <code>$upstream_response_time</code> (and <code>$upstream_header_time</code>) alongside it.

Leaving the FD limit at the default 1024

FDs are cheap; exhaustion is a cliff. Each proxied request uses two, keepalive holds them idle, and the limit is the lower of <code>worker_rlimit_nofile</code> and the system <code>ulimit</code>. If you expect 10,000 connections, set the limit to 65,536 and stop worrying.

Ignoring the kernel listen queue

stub_status cannot see it. When <code>net.core.somaxconn</code> or the <code>listen backlog</code> is too small, connections are dropped before nginx accepts them — zero nginx-log evidence, only client timeouts blamed on the network. Monitor <code>TcpExtListenOverflows</code>.

No per-upstream visibility

When one of five backends degrades, aggregate metrics barely move — healthy servers dilute the signal. Without per-server breakdowns parsed from <code>$upstream_addr</code>, partial failures go undetected until the cascade. Especially dangerous with passive health checks, where real user requests are the probes.

Treating reloads as free

Each <code>nginx -s reload</code> spawns new workers, drains old ones, and adds a brief latency bump. A failed reload does not stop nginx — the old config keeps serving and the drift goes unnoticed. Without <code>worker_shutdown_timeout</code>, old workers on WebSocket/long-poll connections may never exit.

Not monitoring temp-file spooling

When buffers overflow, nginx silently writes bodies to disk, adding latency invisible to <code>$upstream_response_time</code>. The gap between <code>$request_time</code> and <code>$upstream_response_time</code> reveals it, but few teams watch that difference — or the temp-file creation rate.

NGINX runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Connections and worker_connections

▸

File descriptors

▸

Upstream health and backend cascade

▸

HTTP errors, 499s, and latency

▸

Request/response buffers and temp files

▸

SSL/TLS

▸

Rate limiting

▸

Caching

▸

Config, reload, and lifecycle

▸

Dynamic upstream DNS

▸

Kernel listen queue and logging

WHERE TO GO NEXT

Setting up NGINX monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides