NGINX Operations Guides on Netdata

How NGINX actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How NGINX actually works in production: a mental model for operators

NGINX is not a multi-threaded server that spawns a thread per connection. It is an event-driven, non-blocking, single-threaded-per-worker process architecture. If you are debugging a production incident where connections are timing out, memory is climbing, or CPU is pinned, this architecture is the lens through which every symptom must be interpreted.

Most production issues involving NGINX are not NGINX bugs. They are resource accounting problems: file descriptors, connection slots, buffer boundaries, or event loop latency. An operator who understands the internal mechanics can read the signals correctly instead of chasing phantom upstream problems or adding hardware that hits the same limit.

NGINX $request_time vs $upstream_response_time: isolating where latency lives

Mon, 01 Jan 0001 00:00:00 +0000

NGINX $request_time vs $upstream_response_time: isolating where latency lives

P95 $request_time doubles. The default assumption is upstream degradation, so you scale the backend, tune the database, and add instances. The latency barely moves. The bottleneck was a slow mobile client, proxy temp-file disk I/O, or a large request body on a lossy network. This is the most common nginx misdiagnosis.

$request_time measures the full cycle from the first byte read from the client to the last byte sent to the client. It includes reading the request, waiting for the upstream, and writing the response. $upstream_response_time measures only the backend portion, from establishing the upstream connection to receiving the last byte of the response body. The gap between them is where client-side, network, and nginx-internal delays live. To avoid chasing phantom backend problems, log both variables and compare them.

nginx 413 Request Entity Too Large: client_max_body_size explained

Mon, 01 Jan 0001 00:00:00 +0000

nginx 413 Request Entity Too Large: client_max_body_size explained

A 413 Request Entity Too Large after adding client_max_body_size 50m to nginx.conf usually means a more specific context still overrides it, or the upstream application rejects the body after nginx accepts it. The directive applies to http, server, and location blocks. Raising it at the edge only moves the failure deeper if the rest of the stack is not adjusted. This guide covers how the directive inherits, when nginx fires the 413, how proxy_request_buffering changes the failure mode, and why you must verify every hop including the upstream application.

nginx 499 status code: why clients close connections before the response

Mon, 01 Jan 0001 00:00:00 +0000

nginx 499 status code: why clients close connections before the response

Status 499 in nginx access logs means the client closed the TCP connection before nginx finished responding. It is an nginx-specific code that never reaches the client, so it is easy to dismiss. In practice, a 499 surge is an early warning: users or intermediaries abandon requests before upstreams officially time out and before 5xx errors spike. Ignore 499s and you usually see 502s or 504s minutes later.

nginx 500 Internal Server Error: how to diagnose it

Mon, 01 Jan 0001 00:00:00 +0000

nginx 500 Internal Server Error: how to diagnose it

A 500 from nginx tells you something failed in the request path, but not whether the failure originated in your application, FastCGI/uWSGI backend, or nginx itself. When 500s spike during an incident, first determine which side of the nginx boundary is breaking.

Unlike 502 Bad Gateway or 504 Gateway Time-out, which point upstream, a 500 can be an application bug passed through by nginx, a configuration error, a permission failure, or resource exhaustion inside an nginx worker.

NGINX 502 Bad Gateway: Causes And How To Fix It

Mon, 01 Jan 0001 00:00:00 +0000

A 502 Bad Gateway means the upstream server returned an invalid response, refused the connection, or terminated before completing the response. Unlike 504, which signals upstream slowness, 502 means the upstream never produced a valid response or nginx could not reach it.

Start with the error log. A single line like connect() failed (111: Connection refused) tells you the upstream is not listening. A line like upstream prematurely closed connection tells you the backend died mid-request. Match the exact message to the root cause.

nginx 503 Service Temporarily Unavailable: causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

nginx 503 service temporarily unavailable: causes and fixes

A 503 Service Temporarily Unavailable from nginx does not always mean the upstream application is broken. A healthy nginx process returns 503 by design when rate limits reject traffic, or when every backend in an upstream block is unavailable. The same status code covers three different failure paths: intentional throttling, upstream exhaustion, or resource saturation on the nginx host. The fix depends on which path the request took.

NGINX 504 Gateway Time-Out: Causes & Fixes

Mon, 01 Jan 0001 00:00:00 +0000

A 504 Gateway Time-out means nginx reached the upstream but the upstream did not finish its response before proxy_read_timeout expired. The default is 60 seconds. Unlike a 502 Bad Gateway, which means nginx never established a valid upstream connection, a 504 means the connection succeeded but the response did not complete in time.

This guide covers isolating slow upstreams via access log variables, tuning timeouts and retries, and distinguishing 504 from 502.

NGINX access log performance: buffering, sampling, and the event loop

Mon, 01 Jan 0001 00:00:00 +0000

NGINX access log performance: buffering, sampling, and the event loop

Every NGINX worker is a single-threaded event loop. By default, finishing a request triggers an immediate, synchronous write of the access log line. Under normal load the cost is negligible. Under incident conditions, when error rates and log volume spike, that synchronous write becomes a compounding failure: disk I/O stalls the event loop, responses slow down, clients time out, and the resulting errors and abandonments generate even more log lines.

NGINX active connections climbing: reading, writing, waiting explained

Mon, 01 Jan 0001 00:00:00 +0000

NGINX active connections climbing: reading, writing, waiting explained

When operators see Active connections climbing in stub_status, the first instinct is often to add capacity. That instinct is usually wrong. The stub_status module exposes exactly seven metrics, and the most useful of them is the breakdown of active connections into Reading, Writing, and Waiting. The absolute number of active connections is almost meaningless without the ratio between these three states. A server with 10,000 active connections where 9,000 are Waiting is healthy. A server with 500 active connections where 400 are Reading may be under a slowloris-style attack.

NGINX backend cascade failure: when slow upstreams take down everything

Mon, 01 Jan 0001 00:00:00 +0000

NGINX backend cascade failure: when slow upstreams take down everything

Users report timeouts. 502 Bad Gateway and 504 Gateway Time-out responses are climbing, and nginx error logs show upstream timeouts. On the nginx host, CPU and memory are normal, and the master process is alive. The proxy is healthy but out of connections.

This is a backend cascade failure. One slow upstream causes nginx workers to hold connections open while waiting for responses, consuming finite worker_connections slots. As slots fill, new requests cannot be forwarded. Traffic concentrates on the remaining healthy backends, which overload and slow down. Eventually every backend times out or fails health checks, and nginx returns 502/504 to all clients while the proxy process remains up.

nginx connect() failed (111: Connection refused) while connecting to upstream

Mon, 01 Jan 0001 00:00:00 +0000

nginx connect() failed (111: Connection refused) while connecting to upstream

HTTP 502 Bad Gateway and the error connect() failed (111: Connection refused) while connecting to upstream mean nginx reached the upstream IP, but the target port actively refused the TCP connection. The backend is either not running, not listening on the interface nginx expects, or a firewall is blocking the port.

This is distinct from upstream timed out (110: Connection timed out). A timeout means the TCP SYN never received a response, usually because a firewall silently dropped the packet or the host is unreachable. Errno 111 means the network path is open but no process is accepting connections. The error log includes the upstream address, such as upstream: "fastcgi://127.0.0.1:9000". Read that line first to isolate the exact backend.

NGINX connection exhaustion: detection, diagnosis, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

NGINX connection exhaustion: detection, diagnosis, and prevention

Users see connection timeouts while load balancer health checks and the NGINX stub_status endpoint still return HTTP 200. New connections are silently dropped. Connection exhaustion is a cliff-edge failure: once the limit is hit, there is no graceful degradation. Connections are refused at the kernel level, or accepted into the TCP backlog but discarded by NGINX because no worker has a free connection slot.

NGINX DNS resolution failures on dynamic upstreams: 502s and resolver_timeout

Mon, 01 Jan 0001 00:00:00 +0000

NGINX DNS resolution failures on dynamic upstreams: 502s and resolver_timeout

What this means

Intermittent 502 Bad Gateway responses that only hit locations using a variable in proxy_pass, such as proxy_pass http://$backend;, point to dynamic DNS resolution failure. Static upstream locations are unaffected. dig from the host may succeed instantly while nginx logs show 502s with latency spikes clustering at exactly 30 seconds, the default resolver_timeout.

nginx resolves upstream hostnames through two paths.

NGINX dropped connections: the accepts vs handled gap

Mon, 01 Jan 0001 00:00:00 +0000

NGINX dropped connections: the accepts vs handled gap

Users report intermittent connection timeouts. Your HTTP 5xx rate is flat. The error log is quiet. Something is dropping traffic before it ever becomes a request.

On every NGINX instance, the stub_status page exposes two cumulative counters: accepts and handled. When accepts grows faster than handled, NGINX is taking connections from the kernel and then discarding them. This gap is a leading indicator of connection-slot or file-descriptor exhaustion. It often starts increasing minutes before the system hits the hard wall.

NGINX limit_req burst and nodelay tuning: rate limiting without blocking real users

Mon, 01 Jan 0001 00:00:00 +0000

NGINX limit_req burst and nodelay tuning: rate limiting without blocking real users

Most production nginx rate limiting configs fall into two camps: no burst at all, which rejects legitimate traffic during harmless spikes, or burst without nodelay, which queues real users into artificial delays that mimic upstream slowness. Neither is what you want. The limit_req module implements a leaky bucket at millisecond granularity, and the interaction between burst and nodelay determines whether a request is delayed, rejected, or forwarded immediately. Understanding that interaction, and sizing the shared memory zone to match your traffic profile, is the difference between rate limiting that protects upstreams and rate limiting that creates incidents during normal user behavior.

nginx limiting requests, excess -- understanding limit_req rejections

Mon, 01 Jan 0001 00:00:00 +0000

nginx limiting requests, excess – understanding limit_req rejections

When [error] ... limiting requests, excess appears in nginx error logs alongside 503 responses in access logs, determine whether you are under attack, misconfigured, or out of shared memory. The ngx_http_limit_req_module implements a leaky bucket rate limiter. Its interaction with burst, nodelay, and shared memory sizing determines whether you reject malicious traffic, delay legitimate users, or silently stop enforcing limits.

What it is and why it matters

limit_req is nginx’s request-level rate limiter. It uses a shared memory zone, configured via limit_req_zone, to track request rates per key, typically $binary_remote_addr. The zone is mapped into every worker process. When a request arrives, nginx checks the key’s current rate against the configured limit. Depending on burst and nodelay, it delays the request, rejects it, or processes it immediately.

NGINX listen queue overflow: somaxconn, backlog, and silent connection drops

Mon, 01 Jan 0001 00:00:00 +0000

NGINX listen queue overflow: somaxconn, backlog, and silent connection drops

Clients report intermittent connection timeouts. Your load balancer health checks pass. NGINX error logs are clean and access logs show no 5xx spikes. The issue is not in NGINX workers or upstream applications. It is in the kernel accept queue.

When the accept queue fills, the kernel drops new connections silently. NGINX never sees them, so it logs nothing. Evidence is client-side timeouts and the kernel counter TcpExtListenOverflows.

NGINX log disk full: when logging silently stops and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

NGINX log disk full: when logging silently stops and how to recover

During an incident, upstream latency climbs and 5xx errors appear. You run tail -f /var/log/nginx/error.log and the cursor sits there. The last entry is hours old. Requests still return 200s. The server is serving, but it has stopped logging. Disk utilization shows /var/log at 100 percent.

This is the NGINX log disk-full failure mode. The process never signals the client or the operator that it can no longer write diagnostics. Visibility evaporates when it is most needed.

NGINX monitoring checklist: the signals every production server needs

Mon, 01 Jan 0001 00:00:00 +0000

NGINX monitoring checklist: the signals every production server needs

NGINX is an event-driven, single-threaded-per-worker process. Most production failures follow predictable patterns: connection exhaustion, backend cascades, file descriptor limits, or silent kernel-level drops. This article maps the signals that expose those failures into four cumulative maturity levels: Survival, Operational, Mature, and Expert. Use it to audit your current coverage or to justify instrumentation work before the next incident.

Each level adds depth. Survival answers “Is it up?” Operational answers “Is it healthy?” Mature adds leading indicators. Expert adds the signals you instrument after your third postmortem. The tables below list each signal, why it matters, and the threshold that should trigger a response.

NGINX monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

NGINX monitoring maturity model: from survival to expert

nginx exposes exactly seven scalars through stub_status. Latency distributions, upstream health, cache efficiency, and kernel-level drops live in access logs, error logs, or OS counters. Teams that collect only the stub_status numbers assume they have visibility. They do not.

This article defines four monitoring maturity levels. Level 1 tells you if nginx is alive. Level 2 tells you if it is healthy. Level 3 gives you leading indicators of saturation. Level 4 exposes the blind spots that only appear after repeated incidents. Use these levels to audit your current coverage and decide which signals to add next.

nginx no live upstreams while connecting to upstream: what it means

Mon, 01 Jan 0001 00:00:00 +0000

nginx no live upstreams while connecting to upstream: what it means

When nginx logs no live upstreams while connecting to upstream, every server in the affected upstream block is marked unavailable. The proxied request has no eligible backend, so nginx returns 502 Bad Gateway. This is not an nginx defect; it signals that all backends have failed open-source nginx’s passive health checks, or a network partition has made them unreachable from the nginx host.

NGINX old worker processes accumulating after reload: worker_shutdown_timeout

Mon, 01 Jan 0001 00:00:00 +0000

NGINX old worker processes accumulating after reload: worker_shutdown_timeout

After nginx -s reload, worker process count climbs above worker_processes. Memory and file descriptor usage grow. The workers are not crashing; they are old generations waiting for long-lived connections to close.

This pattern is normal in small doses. During a reload, the master keeps old workers alive until active connections drain. Without a shutdown deadline, a single WebSocket, gRPC stream, or long-polling connection can pin an old worker indefinitely. In environments that reload frequently, such as Kubernetes ingress controllers reacting to endpoint changes, accumulation becomes a resource leak that can exhaust memory or file descriptors.

NGINX proxy buffer spill to disk: proxy_buffers and temp file latency

Mon, 01 Jan 0001 00:00:00 +0000

NGINX proxy buffer spill to disk: proxy_buffers and temp file latency

You notice some proxied requests are crawling. Upstream logs show sub-50 ms response times. The network path is clean. The nginx error log is quiet. In the access log, however, $request_time is ten times larger than $upstream_response_time. For large responses, this gap is the signature of proxy buffer spill.

When an upstream response exceeds the memory buffers allocated by proxy_buffers, nginx writes the overflow to a temporary file under proxy_temp_path and reads it back later. Because the log message for this event is emitted at debug level only, the delay is silent. Standard upstream monitoring gives no hint; the delay hides entirely inside nginx.

NGINX proxy buffer tuning: proxy_buffers, proxy_buffer_size, and busy buffers

Mon, 01 Jan 0001 00:00:00 +0000

NGINX proxy buffer tuning: proxy_buffers, proxy_buffer_size, and busy buffers

When proxy_buffering is on (the default), NGINX absorbs the upstream response in memory before sending it to the client. This shields backends from slow clients and enables compression, but three directives control it: proxy_buffer_size for headers, proxy_buffers for the body, and proxy_busy_buffers_size for the in-flight flush window. Misconfiguration causes 502s, silent disk spills, and reload failures.

The defaults are modest: eight body buffers of one memory page each, and one header page (typically 4K or 8K). That works for static sites and small JSON, but it fails for modern workloads: APIs with large JWT tokens in headers, bulk exports returning multi-megabyte JSON, and Server-Sent Events streams. Undersized body buffers spill to disk. Undersized header buffers return 502. Invalid proxy_busy_buffers_size values prevent NGINX from starting or reloading.

NGINX proxy cache hit rate is low: measuring and improving it

Mon, 01 Jan 0001 00:00:00 +0000

NGINX proxy cache hit rate is low: measuring and improving it

Low NGINX proxy cache hit rate shows up as upstream CPU climbing or origin traffic higher than expected. Access logs show MISS and BYPASS where you expect HIT. When caching fails, every request reaches the backend, adding latency and load. The symptom is usually a gradual slide from 85% to 40% over a day, or a collapse to zero after a deployment or restart. Root cause is often configuration drift: a new header, a changed query parameter, or a keys_zone sized for last year’s traffic. Diagnose by measuring which cache status dominates, then trace that status back to the directive or upstream behavior that produces it.

NGINX proxy_cache not caching: why responses bypass the cache

Mon, 01 Jan 0001 00:00:00 +0000

NGINX proxy_cache not caching: why responses bypass the cache

After enabling proxy_cache and defining the cache path, upstreams still take every hit. Access logs show $upstream_cache_status as BYPASS or MISS, hit rate stays near zero, and nothing appears in the error log. NGINX applies a strict request-phase and response-phase decision tree before anything enters the cache. If any condition fails, the response is never stored. The symptom looks like upstream overload, but the root cause is usually a directive, a header, or a missing validity window.

NGINX rate limiting returns 503 not 429: limit_req_status explained

Mon, 01 Jan 0001 00:00:00 +0000

NGINX rate limiting returns 503 not 429: limit_req_status explained

When limit_req rejects a request, nginx returns 503 by default. This is the same code used for genuine capacity exhaustion, so a 503 spike pages the on-call rotation even when the upstream is healthy and the infrastructure is fine. Changing limit_req_status to 429 is one directive, but the implications for alerting, monitoring, and client behavior are not trivial.

What it is and why it matters

The limit_req_status directive sets the HTTP status code returned when limit_req rejects a request. The companion directive limit_conn_status does the same for connection limits enforced by limit_conn. Both default to 503.

nginx recv() failed (104: Connection reset by peer) while reading from upstream

Mon, 01 Jan 0001 00:00:00 +0000

nginx recv() failed (104: Connection reset by peer) while reading from upstream

You tail the nginx error log and see this:

[error] ... recv() failed (104: Connection reset by peer) while reading response header from upstream

The client gets a 502 Bad Gateway. The error is not a configuration syntax problem, and the service was working five minutes ago. This message means the upstream server sent a TCP RST while nginx was mid-read on an upstream connection. The reset originates from the backend or from a network middlebox between nginx and the backend. It does not come from nginx itself, and it does not come from the client.

NGINX reload not applying config: why old workers keep serving

Mon, 01 Jan 0001 00:00:00 +0000

NGINX reload not applying config: why old workers keep serving

You pushed a config change, ran nginx -s reload, and moved on. Hours later, the new certificate is not being served, the updated upstream is not receiving traffic, or the tightened rate limit never took effect. NGINX did not stop running, but the reload never applied. This is the silent rollback: when a reload fails validation, the master keeps the previous configuration active and old workers continue serving. Even when validation passes, old workers can remain alive for hours if long-lived connections prevent them from draining and worker_shutdown_timeout is not set. This guide shows how to confirm the failure, find the root cause, and prevent config drift from going undetected.

NGINX slow requests: from access log to root cause

Mon, 01 Jan 0001 00:00:00 +0000

NGINX slow requests: from access log to root cause

Elevated $request_time in access logs does not mean the upstream is slow. The variable measures the full lifecycle: from reading the first client byte through sending the last response byte. That includes client upload, upstream wait, nginx processing, and client download. Blaming the backend by reflex is the most common nginx latency mistake.

To split the time accurately, confirm your log_format includes $request_time, $upstream_response_time, $upstream_connect_time, and $upstream_header_time.

NGINX slowloris and slow-client attacks: detection and mitigation

Mon, 01 Jan 0001 00:00:00 +0000

NGINX slowloris and slow-client attacks: detection and mitigation

You check stub_status and the Reading count is triple its normal value and not dropping. Requests per second has collapsed to near zero. Active connections are climbing toward worker_connections * worker_processes while worker CPU stays idle. This is not a backend slowdown. It is a slowloris or slow-client attack: connections open faster than they complete, and NGINX waits for data that arrives one byte at a time.

NGINX SSL certificate expired: detection and emergency renewal

Mon, 01 Jan 0001 00:00:00 +0000

NGINX SSL certificate expired: detection and emergency renewal

An expired SSL certificate on NGINX is an immediate outage, not gradual degradation. Browsers and API clients reject the connection at the TLS handshake, often before NGINX logs anything useful. The fix is rarely as simple as running a renewal script again. Verify what is on disk, confirm the running configuration is using it, and force a reload so workers load the new certificate.

NGINX SSL session cache: improving TLS resumption and cutting CPU

Mon, 01 Jan 0001 00:00:00 +0000

NGINX SSL session cache: improving TLS resumption and cutting CPU

TLS handshakes are the most CPU-intensive routine operation in nginx. A worker terminating RSA-2048 TLS can handle only a few hundred full handshakes per second, compared to tens of thousands of plain HTTP requests. In production, every reconnecting client that repeats a full handshake wastes CPU and adds latency. The ssl_session_cache directive exists to eliminate that waste by allowing session resumption across connections. Yet many configurations either omit it, under-size it, or misunderstand how TLS 1.3 changes resumption behavior. This article explains the mechanism, sizing, and the signals that tell you whether your cache is working.

nginx SSL_do_handshake() failed — diagnosing TLS handshake errors

Mon, 01 Jan 0001 00:00:00 +0000

nginx SSL_do_handshake() failed — diagnosing TLS handshake errors

You see [crit] or [error] entries for SSL_do_handshake() failed in /var/log/nginx/error.log. The message might involve a client connecting to nginx, or nginx connecting to an upstream server. These errors mean the TLS negotiation never completed, so no application data was exchanged. The impact ranges from a few browsers showing security warnings to all proxied traffic returning 502 Bad Gateway.

Before chasing certificates, determine which side of a connection is failing. nginx logs two variants: while SSL handshaking to client and while SSL handshaking to upstream. The first affects visitors hitting your server directly. The second affects nginx as a reverse proxy calling a backend over HTTPS. The symptoms, root causes, and fixes are different.

NGINX SSL/TLS handshake CPU saturation: detection and tuning

Mon, 01 Jan 0001 00:00:00 +0000

NGINX SSL/TLS handshake CPU saturation: detection and tuning

NGINX latency climbs while requests per second flatline. Worker processes are pinned near 100% CPU, yet active connections are nowhere near the worker_connections limit. Access logs show fast upstream response times, but $request_time is an order of magnitude larger. The bottleneck is not the network, disk, or backends: it is the TLS handshake.

When workers burn CPU on cryptography, the single-threaded event loop has no time left for request processing. Every new TCP connection that requires a full SSL handshake adds asymmetric crypto workload. If clients do not resume sessions and the connection rate is high, throughput collapses even though the machine has plenty of idle connection slots. This guide shows how to detect, diagnose, and tune for SSL handshake CPU saturation.

NGINX upstream keepalive: eliminating per-request TCP and TLS overhead

Mon, 01 Jan 0001 00:00:00 +0000

NGINX upstream keepalive: eliminating per-request TCP and TLS overhead

Every proxied request that opens a fresh TCP connection burns latency on the handshake and, if the upstream uses HTTPS, on TLS negotiation. At low volume this cost is invisible. At production throughput it becomes a measurable tax on every request, and at extreme scale it can exhaust the kernel’s ephemeral port range and bury the host in TIME_WAIT sockets. For HTTPS upstreams, the CPU cost of TLS handshakes across thousands of requests per second consumes worker cycles that could be spent proxying traffic.

nginx upstream prematurely closed connection while reading response header

Mon, 01 Jan 0001 00:00:00 +0000

nginx upstream prematurely closed connection while reading response header

upstream prematurely closed connection while reading response header from upstream means the upstream server closed the TCP socket while nginx was still reading response headers. This produces a 502 Bad Gateway. Unlike a timeout, the upstream actively terminated the connection.

The root cause is typically on the backend: a crash, worker recycle, request size limit, or stale keepalive connection the backend closed while nginx tried to reuse it. nginx retries the request on another backend only if proxy_next_upstream includes error (the default for idempotent methods). Retries improve availability but do not fix the underlying issue.

nginx upstream sent too big header while reading response header from upstream

Mon, 01 Jan 0001 00:00:00 +0000

nginx upstream sent too big header while reading response header from upstream

The error log contains upstream sent too big header while reading response header from upstream. Clients receive 502 Bad Gateway. This is not an upstream crash or network timeout. It is a hard size limit: an upstream server is sending response headers larger than the fixed buffer nginx allocates for reading them, so nginx aborts the request and returns 502.

nginx upstream timed out (110: Connection timed out) while connecting/reading

Mon, 01 Jan 0001 00:00:00 +0000

nginx upstream timed out (110: Connection timed out) while connecting/reading

upstream timed out (110: Connection timed out) in the nginx error log usually surfaces to clients as a 504 Gateway Timeout. The suffix after the error string tells you which phase failed: connecting, sending, or reading. That phase determines whether you are looking at a dead backend, a network partition, or a retry storm hiding the real problem.

The defaults are unforgiving. proxy_connect_timeout, proxy_send_timeout, and proxy_read_timeout all default to 60 seconds, and proxy_next_upstream implicitly retries on error and timeout. Retries can mask the root cause while exhausting upstream capacity.

NGINX worker_connections and worker_processes: sizing for real traffic

Mon, 01 Jan 0001 00:00:00 +0000

NGINX worker_connections and worker_processes: sizing for real traffic

NGINX defaults leave most CPU cores idle and exhaust quickly under load. The real limit is often the OS file descriptor ceiling, which silently overrides the directive.

Sizing these parameters means understanding the capacity chain: kernel queue, connection slot, file descriptor limit, event loop. This guide provides concrete rules for static and proxy workloads and the signals that reveal when headroom has disappeared.

NGINX worker_rlimit_nofile: setting file descriptor limits correctly

Mon, 01 Jan 0001 00:00:00 +0000

NGINX worker_rlimit_nofile: setting file descriptor limits correctly

nginx file descriptor exhaustion is a silent failure mode. Existing connections continue to be served, but new connections queue in the kernel backlog and are eventually dropped. The client sees a timeout, while nginx error logs may show nothing until the limit is hit.

worker_rlimit_nofile raises the per-worker file descriptor limit above conservative operating system defaults. It does not operate in isolation. It sits inside a layered stack of kernel parameters, systemd service limits, container runtime defaults, and PAM session policies. A value written into nginx.conf is only effective if the master process inherits a hard limit at least as high, and the kernel ceiling permits it. If any layer caps the limit below your intended value, workers inherit that lower ceiling and your tuning is silently ignored.

nginx: a client request body is buffered to a temporary file — what it means

Mon, 01 Jan 0001 00:00:00 +0000

nginx: a client request body is buffered to a temporary file — what it means

You are tailing the nginx error log during a latency investigation and see the line: a client request body is buffered to a temporary file. It is logged at [warn], not [error], so it is easy to ignore. But the message means a request body has exceeded the in-memory buffer and nginx is now writing that data to disk. On a busy reverse proxy or file-upload endpoint, this behavior can add hundreds of milliseconds or seconds of latency before your upstream application receives the payload. It also consumes file descriptors and disk I/O capacity without ever surfacing as a 5xx status code.

nginx: bind() to 0.0.0.0:80 failed (98: Address already in use)

Mon, 01 Jan 0001 00:00:00 +0000

nginx: bind() to 0.0.0.0:80 failed (98: Address already in use)

The error log shows [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) and the master process exits. During a reload, old workers keep running with the previous configuration, so users may not notice immediately. During system boot or a manual start, the service is down.

Error 98 is EADDRINUSE. The nginx master binds listening sockets before forking workers. If the kernel reports port 80 is occupied, nginx cannot start or apply the new configuration. The holder might be a different service, a stale nginx master after a crash, or another nginx instance. Inside the same configuration, conflicting socket options for the same address:port can also trigger the error.

nginx: configuration file test failed - finding the syntax error

Mon, 01 Jan 0001 00:00:00 +0000

nginx: configuration file test failed - finding the syntax error

An nginx -t or nginx -s reload ending with nginx: configuration file /path/to/nginx.conf test failed means the configuration tree is syntactically invalid or references a missing file. The master process rejects the change, so the running server continues with the previous working configuration. That prevents an outage, but your intended change is silently inactive. Read the exact error message, map it to the real source, fix it, and validate with nginx -t before reloading.

nginx: could not build server_names_hash -- server_names_hash_bucket_size

Mon, 01 Jan 0001 00:00:00 +0000

nginx: could not build server_names_hash – server_names_hash_bucket_size

Running nginx -t or nginx -s reload stops with [emerg] could not build the server_names_hash, you should increase server_names_hash_bucket_size: 32. Alternatively, nginx starts but logs [warn] could not build optimal server_names_hash, you should increase either server_names_hash_max_size: 512 or server_names_hash_bucket_size: 64.

This error means your configuration has exceeded a limit in nginx’s server name hashing logic. nginx builds this hash at configuration load time, not at request time. If the build fails, the configuration test fails. If the build succeeds but is suboptimal, nginx starts with degraded lookup performance. Both directives, server_names_hash_bucket_size and server_names_hash_max_size, are valid only inside the http context.

nginx: no resolver defined to resolve - dynamic upstream DNS

Mon, 01 Jan 0001 00:00:00 +0000

nginx: no resolver defined to resolve - dynamic upstream DNS

502 Bad Gateway responses paired with no resolver defined to resolve example.com in the error log mean proxy_pass uses a variable - for example, proxy_pass http://$backend; - and the enclosing context has no resolver directive.

With a literal proxy_pass, nginx resolves the upstream hostname once at startup or reload and caches the result indefinitely. It never queries DNS again until restart or reload. With a variable-based proxy_pass, nginx resolves the hostname at request time through its internal async resolver. Without a resolver directive, the lookup fails immediately and returns 502.

nginx: too many open files - diagnosing file descriptor exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

nginx: too many open files - diagnosing file descriptor exhaustion

After a traffic spike, the error log shows accept4() failed (24: Too many open files), then goes silent. Existing connections still serve, but new ones cannot land.

File descriptor exhaustion is a hard failure. Once the limit is hit, nginx cannot accept new connections, open upstream sockets, or write to the error log. Default OS limits of 1024 are too low for production reverse proxies. Each proxied request consumes at least two FDs, and idle keepalive connections hold them indefinitely. The effective limit is the lower of worker_rlimit_nofile and the OS hard limit enforced by systemd or the container runtime.

nginx: worker_connections are not enough — causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

nginx: worker_connections are not enough — causes and fixes

Your error log shows worker_connections are not enough while connecting to upstream. New clients time out while existing connections may still work. This is a hard capacity cliff: once a worker exhausts its connection slots, it cannot accept new connections until a slot frees. The default limit is 512 per worker, not 1024, and in reverse-proxy mode each request consumes at least two slots. Raising the number is often the first reaction, but if a slow backend is holding connections open, the slots will fill again no matter how high you set the limit.