HAProxy counter resets on reload: phantom drops and spikes in your metrics

Your HAProxy 5xx dashboard just showed a cliff-edge drop to zero, then a vertical spike an hour later. No user complaints. No backend incidents. No deploys. If this pattern repeats on a schedule, or right after config changes, you are almost certainly looking at counter resets from HAProxy reloads, not real traffic events.

Every time HAProxy reloads its configuration, a new worker process starts. That new process has its own in-memory statistics, and every cumulative counter starts at zero: hrsp_5xx, econ, eresp, bin, bout, wretr, wredis, chkfail, the SSL counters, all of them. A reload is a new process for stats purposes, not a continuation of the old one.

Any monitoring pipeline that computes rates from these counters without handling resets will produce garbage at reload boundaries: phantom negative rates, phantom error spikes, false alerts, or a real incident hidden inside the noise. This is a monitoring configuration problem, not an HAProxy bug. HAProxy has always worked this way.

This article covers what actually resets, how to confirm a reset caused a given graph artifact, and how to make collection, rate math, and alerting reset-aware.

What this means

HAProxy keeps its statistics in process memory. There is no persistence layer for the classic counters. When you run systemctl reload haproxy (or send the equivalent signal), the running process does not mutate its config in place. A new process starts with the new configuration, the old process enters soft-stop and drains its existing connections, and the new process begins serving with a completely fresh set of counters.

Two consequences:

1. Counters go to zero on the new process. A monitoring system scraping hrsp_5xx sees something like: 15,432, 15,471, 15,509, 0, 3, 9. Naive delta math reads the 15,509 to 0 transition as either a huge negative rate (clamped or wrapped into a huge positive rate, depending on the tooling) or as a discontinuity that corrupts the rate window.

2. The old process becomes invisible. After the new process takes over the stats socket, the old draining process’s counters stop updating in most collection setups. Any traffic still flowing through the old worker is unmeasured until it exits.

The artifacts are predictable:

• Phantom drops. The cumulative error graph falls off a cliff to zero. If your alert logic is “5xx total decreased” or your SLO burn math assumes monotonic counters, it misfires.
• Phantom spikes. Some rate computations treat a counter decrease as a wrap-around and compute an enormous positive delta. One reload can produce a single data point that dwarfs real traffic.
• Masked real events. If a genuine error burst happens in the same scrape window as a reload, the reset math can swallow it entirely. The alert that should have fired did not.
• Broken averages. Rolling-window averages (qtime, ctime, rtime, ttime) restart with no samples on the new process. Instantaneous rate fields (rate, req_rate) also restart. Baselines computed across a reload boundary are unreliable.

In environments with frequent automated reloads (Kubernetes ingress controllers, service discovery re-rendering config on every backend change), this can happen dozens of times a day, and every reload is a chance for a false page or a missed one.

Common causes

The distinguishing test for most of these is Uptime_sec from show info. A reload or restart resets it. A manual clear counters does not.

Quick checks

All of these are read-only.

# 1. Current uptime of the active worker. A small value right after a
#    graph artifact confirms a reload or restart caused it.
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(Uptime_sec|Uptime):"

# 2. Is the process you are scraping in soft-stop (draining)?
#    Stopping: 0 = normal, 1 = soft-stop in progress.
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^Stopping:"

# 3. How many haproxy processes are running? More than the expected
#    master plus one worker means a reload just happened or old
#    workers are lingering.
pgrep -c haproxy

# 4. Current cumulative counters for a suspect proxy. Note the values
#    now and compare across the next reload. Field positions are the
#    standard show stat CSV order.
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{print $1"/"$2": 5xx="$44" econ="$14" bin="$9" bout="$10}'

# 5. When did the current process actually start?
ps -o pid,lstart,cmd -C haproxy

Check 1 is the single most useful one. If Uptime_sec is 45 seconds and your dashboard shows a “5xx collapse” 45 seconds ago, you have your answer.

How to diagnose it

When a graph shows an impossible drop or spike in an HAProxy cumulative metric, work through this sequence before treating it as a real event.

1. Overlay Uptime_sec on the artifact. If the metric discontinuity lines up with an uptime drop to near zero, it was a reload or restart. Stop investigating the traffic; start investigating why the reload happened.

2. Check whether the artifact shape matches reset math. A real error spike rises over several samples and decays. A reset artifact is a single-sample discontinuity: cliff to zero, or one absurdly large delta. If your graph shows exactly one anomalous point at the reload instant and normal values on both sides, that is reset math, not traffic.

3. Correlate with other cumulative counters. A real 5xx burst shows up in hrsp_5xx but leaves bin/bout and req_tot growing smoothly. A reload zeroes all of them simultaneously. If every cumulative counter for the proxy broke at the same timestamp, it was a process event.

4. Rule out a restart vs a reload. A graceful reload keeps existing connections alive on the draining worker. A restart drops them. Check the system journal and pgrep -c haproxy history. If connections were reset and there was a service gap, treat it as a restart incident and review why it restarted.

5. Check for a reload storm. If resets are frequent, count processes and look for Stopping: 1 lingering on old workers. Frequent reloads from service discovery plus no hard-stop-after means old workers can hold FDs and memory while draining. That is a separate operational problem, but it also multiplies your counter-reset artifacts.

6. Only then look at the traffic. If uptime is old, no reload occurred, and the counters still moved strangely, check for a manual clear counters (Runtime API audit) or a tooling bug in your collector.

flowchart TD
  A[Graph shows drop or spike in cumulative counter] --> B{Uptime_sec dropped at same time?}
  B -- Yes --> C{Multiple PIDs or Stopping: 1 lingering?}
  C -- Yes --> D[Reload storm: fix reload frequency and hard-stop-after]
  C -- No --> E[Single reload: fix monitoring rate math]
  B -- No --> F{All counters moved together?}
  F -- Yes --> G[Manual clear counters or collector bug]
  F -- No --> H[Real traffic event: investigate service]

Metrics and signals to monitor

Scope note: everything in the stats CSV and show info that is cumulative is per-process. Instantaneous gauges (scur, qcur) survive conceptually but reflect only the new process after a reload, which is also a discontinuity if you were mid-incident.

Fixes

The fix is on the monitoring side. Grouped by approach.

Use reset-aware rate math

The core fix: compute rates with a function that treats a counter decrease as a reset, not as a negative delta or a wrap.

• In Prometheus, rate() and increase() already handle counter resets by discarding the decreasing sample. If you hand-rolled delta queries, switch to these.
• In custom collectors: when current < previous, count only current as the delta for that interval (assume the counter started at zero), or discard the interval entirely and mark it.
• Never alert on the raw cumulative value of an HAProxy counter. Alert on reset-aware rates expressed as ratios (for example, 5xx rate as a percentage of req_rate), which are naturally bounded.

Tradeoff: reset-aware rate math discards or approximates the reload interval. If reloads are frequent and your scrape interval is long, real errors in the pre-reload window are undercounted. Shorten the scrape interval relative to your reload frequency, or gate alerts as below.

Detect resets explicitly and gate alerts

Even with correct rate math, the window around a reload is untrustworthy: first samples are empty, averages have no history, and the old worker’s traffic is invisible. Use the reset detector the platform gives you:

• Track Uptime_sec as a metric. A drop between scrapes means a reload or restart happened. Suppress or annotate alerts whose evaluation window crosses that drop.
• Use Uptime_sec as an alert precondition. The standard pattern is to require Uptime_sec above a threshold (for example, 600 seconds) before rate-based alerts can fire. This eliminates cold-start and post-reload false positives in one move.
• Track the Stopping flag. It tells you a drain is in progress, which explains both the reset and any temporary measurement gap.

Reduce how often resets happen

Every reload you eliminate is a measurement discontinuity you do not have to handle.

• Batch configuration changes instead of reloading on every service discovery event.
• If you run a controller or templating system that reloads on every backend change, add debouncing or change aggregation.
• Set hard-stop-after so draining workers actually exit; lingering soft-stop processes extend the window where two processes exist and one of them is invisible to your collector.

Tradeoff: less frequent reloads mean slower propagation of legitimate config and backend changes. For dynamic environments, find the batching interval that balances convergence speed against metric continuity.

Persist counters across reloads (HAProxy 3.0 and later)

HAProxy 3.0 added a counter persistence mechanism for the first time:

• The Runtime API command dump stats-file writes current proxy counters to a file.
• The global directive stats-file preloads those counters into the next process at startup, so the new worker resumes from the old values instead of zero.
• Only proxy counters (frontends, backends, servers, listeners) are covered, and objects need a guid assigned for the state to match across processes.

This is best-effort, not exact. Connections still draining on the old worker when you dump will lose their subsequent counter increments, and there are known cases where preloading produces small counter decreases rather than a clean continuation, which poorly-behaved rate math can still misread. Test it against your monitoring pipeline before relying on it, and keep the Uptime_sec gating in place regardless. HAProxy 3.3 also added an experimental shared-memory variant (shm-stats-file) and per-field persistence flags in show stat output; treat these as experimental until they stabilize for your version.

Fix the dashboards, not just the alerts

Even with alerting fixed, dashboards that render raw cumulative counters will keep showing cliffs at every reload, and someone will misread one during an incident. Convert panels to reset-aware rates, and mark reload events (from Uptime_sec drops) as annotations so the discontinuities have a visible explanation.

Prevention

• Reset-aware rates everywhere. No HAProxy cumulative counter should ever feed a raw-value alert or a naive delta. Use non-negative derivative / reset-aware rate functions.
• Uptime_sec gating on rate alerts. Require a minimum process age before rate-based alerts evaluate. This one condition removes most false pages from reloads, restarts, and cold starts.
• Reload annotations. Emit an event or annotation on every Uptime_sec drop so graph discontinuities are explainable at a glance.
• Reload hygiene. Batch config changes, debounce discovery-driven reloads, set hard-stop-after, and alert on process count and lingering Stopping: 1 to catch reload storms.
• Ratio-based thresholds. Alert on 5xx as a percentage of request rate, not absolute counts. Ratios are bounded and survive traffic-scale differences, and they behave better across resets.
• Scrape interval shorter than reload interval. If reloads happen every few minutes and you scrape every 60 seconds, your rate math is structurally unreliable. Fix the interval, the reload frequency, or both.

How Netdata helps

• Netdata collects HAProxy stats through the socket and CSV interfaces and computes rates with counter-reset-aware math, so reload zeroing does not turn into phantom negative or wrapped spikes on dashboards.
• Uptime_sec is collected alongside the counters, which makes it possible to see the exact reload moment overlaid with the discontinuity it caused.
• Cumulative error counters (hrsp_5xx, econ, eresp, retries and redispatches) are charted as rates, so a reset appears as a gap or annotation rather than a fake traffic event.
• The Stopping flag and process-level visibility surface lingering draining workers, which is the difference between a normal reload and a reload storm.
• Per-second collection granularity shortens the corrupted window around each reload compared to minute-level scraping, so less real signal is lost at the boundary.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy connection reuse dropping: http-reuse, pooling, and handshake overhead
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy ephemeral port exhaustion: TIME_WAIT and backend connection churn
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors