NGINX proxy buffer spill to disk: proxy_buffers and temp file latency

You notice some proxied requests are crawling. Upstream logs show sub-50 ms response times. The network path is clean. The nginx error log is quiet. In the access log, however, $request_time is ten times larger than $upstream_response_time. For large responses, this gap is the signature of proxy buffer spill.

When an upstream response exceeds the memory buffers allocated by proxy_buffers, nginx writes the overflow to a temporary file under proxy_temp_path and reads it back later. Because the log message for this event is emitted at debug level only, the delay is silent. Standard upstream monitoring gives no hint; the delay hides entirely inside nginx.

What this means

With proxy_buffering on (the default), nginx eagerly reads the upstream response into memory. proxy_buffer_size sets the buffer for the response headers. proxy_buffers sets the number and size of additional buffers for the body. The default is typically 8 buffers of one memory page each (4 kB or 8 kB per buffer, depending on architecture).

Once the response body fills the available buffers, nginx writes the remainder to disk. It opens a temporary file in proxy_temp_path (default: a proxy_temp directory relative to the nginx prefix) and flushes excess data there. Later, the worker reads that file to send the response to the client. Each spilled response incurs at least two disk I/O operations: a write during the upstream receive, and a read during the client send. Because the worker must complete the write before reusing the buffer for more upstream data, slow writes stall the read. If the disk is slow or contended, that I/O dominates $request_time.

The proxy_busy_buffers_size directive limits how much data can be held in buffers actively being sent to the client while the upstream response continues. While this tunes the overlap between receiving and sending, it does not prevent spill.

The operational reality is that the message indicating a response is buffered to a temporary file appears only at debug level. At the default error level, there is no log entry. The only symptoms are elevated $request_time, a gap above $upstream_response_time, and disk I/O on the partition hosting the temp directory.

flowchart LR
    U[Upstream response] --> B{Fits in proxy_buffers?}
    B -->|Yes| M[Serve from memory]
    B -->|No| T[Write to proxy_temp_path]
    T --> R[Read temp file]
    M --> C[Send to client]
    R --> C

Common causes

Quick checks

Run these read-only checks to confirm buffer spill and locate the temp path.

# Verify current proxy buffer configuration
nginx -T 2>/dev/null | grep -E 'proxy_buffer|proxy_temp|proxy_max_temp'

# Locate the configured proxy_temp_path
nginx -T 2>/dev/null | grep proxy_temp_path

# Look for the header-overflow error signature
grep -c "upstream sent too big header" /var/log/nginx/error.log

# Inspect access log for the latency gap.
# Adjust field indices to match your log_format.
# This example assumes $request_time and $upstream_response_time are the last two fields.
tail -n 10000 /var/log/nginx/access.log | \
  awk '{
    rt=$(NF-1); urt=$NF;
    if (rt > urt * 5 && rt > 0.2) print $0
  }' | head -20

If proxy_temp_path is not explicitly set, the directory is relative to the nginx prefix. Check your distribution’s default or inspect the prefix with nginx -V 2>&1 | grep prefix.

How to diagnose it

1. Confirm the latency gap in access logs. Compare $request_time and $upstream_response_time. A large, persistent delta where upstream time is normal but total time is high points to nginx-internal delay, most often disk I/O from temp file spill.

2. Correlate with response size. Buffer spill only happens when the response body exceeds available memory buffers. Check $body_bytes_sent in the access log. Endpoints with responses larger than the total proxy_buffers space are the prime suspects.

3. Check the temp directory. Look for files under proxy_temp_path during peak traffic. Files may be short-lived; run find <proxy_temp_path> -mmin -1 in a loop while reproducing the slow request. If your distribution places temp files under the nginx prefix, adjust the path accordingly.

4. Verify buffer sizing. Run nginx -T | grep -E 'proxy_buffers|proxy_buffer_size'. Calculate total memory buffer space: proxy_buffer_size + (proxy_buffers count * size). For example, with proxy_buffer_size 4k and proxy_buffers 8 4k, the total is 36 kB. If your P95 response body exceeds this total, spill is guaranteed for that traffic.

5. Check disk latency for the temp partition. Use iostat -x 1 or disk latency metrics for the filesystem hosting proxy_temp_path. If average write latency exceeds single-digit milliseconds, the disk is the bottleneck.

6. Confirm log silence. Check the error log at default level. Body buffer spill produces no message. If you see “upstream sent too big header,” that is a separate proxy_buffer_size issue, not body buffer overflow.

Metrics and signals to monitor

Fixes

Increase proxy_buffers

Size the directive so that typical large responses fit in memory. The directive takes a count and a size, for example proxy_buffers 16 16k;. Total memory per request for the body is count * size, plus proxy_buffer_size for the headers.

Tradeoff: each active proxied connection allocates these buffers. Raising counts increases per-connection memory, which multiplies across all active connections. Calculate total memory impact before increasing. With proxy_buffers 16 16k, body buffers alone consume 256 kB per request; at 1,000 concurrent connections that is 256 MB.

Increase proxy_buffer_size for headers

If the error log shows “upstream sent too big header while reading response header from upstream,” raise proxy_buffer_size. This is distinct from body buffering but produces similar 502 symptoms.

Relocate proxy_temp_path to tmpfs

Mount the temp directory on tmpfs to eliminate physical disk latency.

Tradeoff: tmpfs consumes RAM. The space needed scales with concurrent large responses and proxy_max_temp_file_size. If nginx spills gigabytes concurrently, you must size tmpfs accordingly or risk memory pressure and OOM.

Move proxy_temp_path to faster storage

If tmpfs is not viable, place the directory on local SSD or NVMe rather than network-attached or spinning disk. This does not eliminate the I/O but reduces the latency penalty.

Disable disk buffering with proxy_max_temp_file_size

Set proxy_max_temp_file_size 0; to prevent writing temporary files entirely. nginx then streams the response to the client synchronously.

Tradeoff: the upstream connection stays open until the client finishes receiving the response. Slow clients hold upstream connections longer, which can exhaust upstream keepalive pools or backend connection limits. Use this only when you can tolerate the upstream retention cost.

Enable proxy_cache for cacheable large responses

For responses that do not change per request, caching avoids both repeated upstream fetches and repeated temp file creation. Ensure the cache key and TTL match your workload.

Prevention

• Size proxy_buffers from access-log percentiles. The default memory page buffers handle small payloads. If your P95 response body exceeds the total buffer space, spill is inevitable.
• Monitor the request-to-upstream latency gap. A sustained delta between $request_time and $upstream_response_time is the earliest signal of internal buffering delay.
• Place proxy_temp_path on tmpfs or fast local storage. This removes disk seek latency from the critical path when spill does occur.
• Review large response patterns. Upstream endpoints returning multi-megabyte bodies to every request may need pagination, compression, or caching instead of larger buffers.
• Watch for header buffer errors. “upstream sent too big header” is a distinct but related failure that shares the same 502 symptom; keep proxy_buffer_size sized for your header footprint.

How Netdata helps

• Surfaces the $request_time to $upstream_response_time gap in nginx percentiles, revealing temp file latency.
• Monitors disk latency for the partition hosting proxy_temp_path, correlating I/O spikes with request latency.
• Tracks nginx active connections and file descriptor usage per worker, showing when buffer-related memory or FD pressure builds.
• Alerts on sustained nginx 5xx rates and upstream latency deviations.

Related guides

• How NGINX actually works in production: a mental model for operators
• nginx 499 status code: why clients close connections before the response
• nginx 500 Internal Server Error: how to diagnose it
• nginx 502 Bad Gateway: causes and how to fix it
• nginx 503 Service Temporarily Unavailable: causes and fixes
• nginx 504 Gateway Time-out: causes and fixes
• NGINX active connections climbing: reading, writing, waiting explained
• NGINX backend cascade failure: when slow upstreams take down everything
• nginx connect() failed (111: Connection refused) while connecting to upstream
• NGINX connection exhaustion: detection, diagnosis, and prevention
• NGINX dropped connections: the accepts vs handled gap
• NGINX monitoring checklist: the signals every production server needs