HAProxy DroppedLogs: losing log lines exactly when you need them

You are mid-incident. Traffic is spiking, 5xx rates are climbing, and you go to the HAProxy logs for per-request timing and termination codes. The logs are not there. Whole seconds of traffic are missing, right at the peak. Then you notice DroppedLogs in show info has been incrementing for hours and nobody was watching it.

DroppedLogs counts log messages HAProxy tried to emit but could not deliver. The cause is almost never HAProxy itself: it is the log pipeline downstream. A syslog daemon that cannot keep up, a socket buffer that overflowed, a log target that stalled. There is no error in the HAProxy log, because the line that would have told you is one of the ones that got dropped.

The failure mode has a cruel symmetry: log volume scales with traffic, so the pipeline is most likely to saturate exactly when traffic is highest, which is when you need per-request logs most. Any nonzero increment of DroppedLogs deserves a ticket. Not because dropped logs hurt traffic (they do not), but because they destroy your ability to diagnose the next problem.

What this means

HAProxy emits a log line when a proxied session ends. Under load this is thousands of lines per second, going out over whatever log target you configured: a local syslog socket, a UDP syslog endpoint, a file descriptor, or a ring buffer. (In HTTP mode the log fires at connection end, so one line may cover several keep-alive requests; HTTP/2 logs per stream. )

When the delivery path cannot absorb the volume, HAProxy does not block the event loop waiting for the log sink. It drops the message and increments DroppedLogs. That is the right design for a proxy (logging must never stall traffic), but it means the log sink’s capacity problem becomes your observability problem.

flowchart LR
  S[Sessions close] --> E[HAProxy emits log line]
  E --> B{Log sink accepts?}
  B -->|yes| L[syslog / file / ring]
  B -->|no, buffer full or sink slow| D[DroppedLogs counter +1]
  L --> W[written to disk or forwarded]
  D -.->|silent: no log line| X[message lost forever]

Two properties matter operationally:

  • Dropped messages are gone. HAProxy does not retry or spool them. The counter is the only evidence they existed.
  • The counter only sees local delivery failures. If HAProxy hands a UDP datagram to the kernel and the network loses it, or the remote syslog drops it after receipt, DroppedLogs does not move. The counter reflects “HAProxy could not send,” not “the log line made it to disk.”

One nuance: the HAProxy documentation ties this drop-and-count behavior to unbuffered file descriptor targets (fd@<n>, stdout, stderr), where a failed writev() increments the counter. With UDP syslog targets, the kernel socket buffer can also overflow silently. Operators have reported the counter moving with local syslog socket targets too, so treat any increment as real regardless of target type, but do not assume a zero counter means every line reached your log store.

Common causes

CauseWhat it looks likeFirst thing to check
Syslog daemon too slow or stalledDroppedLogs increments during traffic peaks, recovers off-peakIs rsyslog/syslog-ng alive and keeping up? Check its own impstats or queue depth
Socket receive buffer too smallDrops correlate tightly with log line rate, syslog process looks healthyKernel receive buffer sizes and the syslog listener’s buffer tuning
Syslog daemon restart or socket recreationA burst of drops at a specific timestamp, then back to zerosyslog daemon logs, unit restart history
Log volume growth from config changeDrops started right after enabling more verbose logging (more fields, logging health checks, debug level)Recent config diffs for log-format, option httplog, option dontlognull
Downstream of syslog is blockedrsyslog queue grows because disk or remote forwarder is slow, backpressure reaches the socketrsyslog queue statistics, disk latency on the log volume
Traffic surge without headroomDrops only during bursts (deploys, cron fans, attacks)Correlate DroppedLogs delta with SessRate (show info) and req_rate (CSV stats)

Quick checks

All of these are read-only. Adjust the socket path if your build puts it elsewhere (commonly /var/lib/haproxy/stats).

# 1. Read the counter itself
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^DroppedLogs:"

# 2. Take two samples 60s apart to get a drop rate, not just a lifetime count
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^DroppedLogs:"
sleep 60
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^DroppedLogs:"

# 3. Confirm current traffic and log volume pressure
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(SessRate|ConnRate|CurrConns|Uptime_sec):"

# 4. See what log targets HAProxy is configured with
grep -E "^\s*log " /etc/haproxy/haproxy.cfg

# 5. Is the syslog daemon running and has it restarted recently?
systemctl status rsyslog --no-pager 2>/dev/null | head -5 || systemctl status syslog-ng --no-pager | head -5

# 6. Kernel UDP receive buffer defaults (per-socket limits the syslog listener inherits)
sysctl net.core.rmem_default net.core.rmem_max

# 7. Where are the gaps? Show per-second line counts, lowest first
awk '{print substr($0,1,15)}' /var/log/haproxy.log | uniq -c | sort -n | head

Notes on interpretation:

  • Uptime matters. DroppedLogs is a process counter. A large value with a long Uptime_sec may be ancient history; the 60-second delta is what tells you whether you are dropping now.
  • Check the syslog daemon’s restart history. A syslog restart removes and recreates the local socket or briefly stops receiving; lines sent in that window are lost. If drop bursts line up with package updates or logrotate-triggered restarts, that is your cause.
  • Check 7 finds thin seconds, not empty ones. Seconds with zero lines never appear in the output because they produce no input lines. During a busy period, look for counts of 1 or 2 amid thousands; that is your gap.

How to diagnose it

  1. Establish the drop rate. Two samples of DroppedLogs 60 seconds apart. Zero means the incident is historical; positive means the pipeline is failing now. Express it as drops per second and compare to your log line rate, which tracks SessRate (one line per session in HTTP mode).

  2. Identify the configured log path. Look at every log line in the global and defaults sections. The three shapes you will find: a Unix socket (/dev/log or an rsyslog-created socket), UDP to a local or remote host:port, or a file descriptor / ring target. The fix is different for each.

  3. Check the syslog daemon end of the pipe. If you log to rsyslog, look at its internal queue statistics (impstats) if enabled: a growing main queue or a discarded-message counter on the imuxsock/imudp input is the smoking gun. If rsyslog forwards to a remote destination, check whether that downstream is slow, because rsyslog backpressure propagates back to the socket HAProxy writes to.

  4. Time-correlate with restarts and config changes. Compare drop onset with syslog restarts, HAProxy reloads, logrotate runs, and any change that increased per-line size or lines per session. A custom log-format with extra sample fetches can make lines substantially longer; longer lines fill buffers faster.

  5. Check buffer sizing against peak line rate. For a UDP or Unix socket target, the kernel receive buffer is the shock absorber between HAProxy’s burst and the syslog daemon’s read loop. If the burst exceeds buffer bytes divided by average line size, you drop. Defaults are often far too small for thousands of lines per second.

  6. Rule out the network path (for remote syslog). Remember the counter’s blind spot: datagrams HAProxy sent successfully can still be lost in the network or dropped by the receiving host’s own socket buffer. If DroppedLogs is zero but lines are missing at the destination, the loss is downstream. Compare line counts at the receiver against HAProxy’s session counters for the same window.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
DroppedLogs (show info)The only signal HAProxy gives you that log delivery failedAny increment; alert on delta > 0
SessRate (show info) / req_rate (CSV stats)Approximates log line rate; drops should correlate with peaksDrop rate rising with request rate means no headroom
Uptime_sec (show info)Lets you interpret the counter and detect reloads that reset baselinesCounter interpretation without uptime is meaningless
Syslog daemon input queue / discard countersThe receiving end’s view; confirms which side is saturatedQueue persistently nonzero, discards incrementing
Kernel UDP errors (/proc/net/snmp RcvbufErrors, NoPorts)Receiver-side socket buffer overflows invisible to HAProxyRcvbufErrors incrementing during peaks
Log volume disk write latency / utilizationSlow disk backs up the syslog daemon, which backs up the socketWrite latency spikes correlated with drop bursts

Fixes

Grouped by cause. None of these require restarting HAProxy.

Syslog daemon cannot keep up

Make the receiver faster before touching HAProxy. Typical levers: move rsyslog’s main queue to a disk-assisted or purely in-memory queue sized for your peak line rate, disable expensive per-message processing (deep parsing, heavy templates) on the HAProxy input, and write logs with asynchronous flushing if filesystem latency is the bottleneck. If rsyslog forwards to a remote collector, decouple the forward with a local action queue so downstream slowness cannot backpressure the input socket.

Tradeoff: in-memory queues lose buffered messages if the daemon crashes. That is still better than losing them continuously at the socket.

Socket buffer too small

Increase the receive buffer on the syslog input socket and raise the kernel ceilings (net.core.rmem_max) so the daemon is allowed to request a big buffer. Note that not every input module exposes buffer sizing: rsyslog’s imudp does, imuxsock support is limited. Size the buffer to absorb your worst realistic burst: peak lines per second multiplied by average line size multiplied by the seconds of stall you want to survive.

Tradeoff: bigger buffers hide brief stalls but cannot fix a receiver that is chronically slower than the line rate. If the daemon reads slower than HAProxy writes on average, no buffer size saves you.

Syslog restarts dropping logs

If drops align with syslog restarts, reduce restart frequency (config management churn is the usual cause) and make restarts fast. On the HAProxy side, a more durable fix is to change the transport.

Move to a buffered or reliable transport

Two structural options:

  • Ring buffers. HAProxy can log to an internal ring buffer (ring@<name>) and have a log server drain it. This decouples emission from delivery: short downstream stalls are absorbed by the ring instead of dropping messages at emit time. The tradeoff is that a full ring overwrites old messages rather than counting drops, so you trade silent loss at the socket for silent loss in the ring unless you also watch the ring. Rings currently expose little in the way of operational statistics, so visibility into ring overflow is limited.
  • TCP instead of UDP for the hop to the log collector, either from HAProxy’s log forwarding or from the local syslog daemon onward. TCP gives you backpressure and retransmission instead of fire-and-forget. Load tests of ring-based log forwarding have reported near-total message loss over UDP backends under load and zero loss over TCP, which matches what the transports promise. The tradeoff is that backpressure now has somewhere to propagate; make sure the receiving end can genuinely keep up.

Reduce log volume

Sometimes the honest fix is emitting fewer or smaller lines: drop health-check and monitoring-probe traffic from logs with dontlognull or ACL-based no log rules, trim sample fetches from a bloated custom log-format, and stop logging at debug/info globally. Every byte you do not emit is buffer you do not need.

Prevention

  • Alert on the delta. DroppedLogs should be zero forever. A delta greater than zero over any 5-minute window opens a ticket. This is cheap and catches the failure before the incident where you needed the logs.
  • Watch the receiver, not just the sender. HAProxy’s counter cannot see network loss or receiver-side socket overflows. Monitor the syslog daemon’s discard counters and the kernel’s UDP receive buffer errors on the receiving host, or your zero-drop signal is incomplete.
  • Load-test the log path, not just the proxy. When you capacity-test HAProxy, verify the log pipeline at the same line rate. A log sink that handles average traffic but not 3x peak is a time bomb.
  • Treat log verbosity changes as capacity changes. A config diff that adds fields to log-format or removes a dontlog rule changes bytes per second through the pipeline. Review it like a traffic change.
  • Survive syslog restarts. Prefer a log path that buffers across a syslog restart (local ring, persistent queue) so routine maintenance does not punch holes in your records.

How Netdata helps

  • Netdata’s HAProxy collector reads show info continuously, so DroppedLogs becomes a per-second time series instead of a number you remember to check during an incident. A nonzero delta is visible immediately and alertable.
  • Request rate, session rate, and connection counts are collected at the same frequency, so you can confirm the classic signature on one dashboard: drop rate tracking SessRate at peak, which separates “pipeline has no headroom” from “pipeline is broken.”
  • Counter resets after reloads show up as discontinuities; correlating with Uptime_sec keeps you from misreading a fresh process as a cured problem.
  • Netdata also monitors the syslog side of the pipe when the daemon runs on the same host: process liveness, CPU, disk write latency on the log volume, and UDP error counters, which is where most root causes live.
  • During an incident, having the drop timeline next to the 5xx and latency timeline tells you quickly whether your log gaps were caused by the same event you are investigating.