HAProxy stick-table overflow: rate limiting that silently stops working

Your HAProxy config has rate limiting. It has session persistence. Both are enforced by stick tables, and both can stop working without an error counter, a log line, or a page.

Stick tables are fixed-size, in-memory key-value stores. When a table reaches its configured size, HAProxy has two possible behaviors and neither one alerts you. With the default configuration, HAProxy flushes expired entries to make room for new ones. If the table is full of entries that have not yet expired, new entries are refused. With the nopurge option, eviction is disabled entirely and new entries are silently rejected. In both paths, whatever the table was doing (rate limiting, sticky sessions, abuse tracking) degrades or stops for affected clients.

The way most teams find out: a DDoS or credential-stuffing wave succeeds despite rate limiting being configured. The config was correct. The table was full. This article covers how to confirm overflow, how the two overflow modes differ, and how to size and monitor tables so it never gets that far.

What this means

A stick table entry is created the first time HAProxy sees a key (typically a source IP via track-sc0 or http-request track-sc0 src). That entry carries counters: http_req_rate, conn_rate, gpc0, and so on. Every subsequent request from that key updates the entry, and ACLs read the counters to decide whether to deny, throttle, or pin the session.

When the table is full:

  • Default (no nopurge): HAProxy flushes a few of the oldest expired entries to release space. This is silent: no counter, no log. If the table is being filled faster than entries expire (for example, a botnet rotating through unique source IPs with a long expire), eviction happens continuously and tracking effectively resets for evicted clients. Their rate counters vanish and they start fresh, which defeats the rate limit.
  • With nopurge: No eviction occurs. New keys are refused. A new client’s track-sc0 action fails silently: no entry exists, so no counters exist, so the rate-limit ACLs never trigger for that client. Rate limiting never activates for new clients at all.

Neither path increments dreq, dresp, or any other CSV stat. The stats page and the show stat CSV export contain no stick-table fields at all. The only visibility is the runtime API command show table.

flowchart TD
    A[New key arrives] --> B{Table full?}
    B -- No --> C[Entry created, counters tracked]
    B -- Yes --> D{nopurge set?}
    D -- No --> E{Expired entries exist?}
    E -- Yes --> F[Oldest expired entries evicted - silent]
    E -- No --> G[New entry refused - silent]
    D -- Yes --> G
    F --> H[Evicted client counters reset, rate limit defeated]
    G --> I[New clients never tracked, rate limit never activates]

One more multiplier: if you replicate tables via the peers subsystem, a full table on one node implies the peers are full too, because peers replicate the full table state. You cannot dodge the problem by failing over.

Common causes

CauseWhat it looks likeFirst thing to check
Table size too small for real client cardinalityused pinned at or near size in show tableCompare used vs size; estimate unique keys per expire window
expire too long for the traffic profileTable fills with stale entries that outlive their usefulnessEntry timestamps in show table <name> output; ratio of new-key rate to expire window
DDoS or scan with many unique source IPsSudden fill coinciding with a session-rate spikeFrontend ConnRate and SessRate from show info; source IP distribution in show table <name>
nopurge on a table sized for average, not peakRate limiting never engages for new clients during burstsTable config; used at 100% while abuse traffic continues
Expired entries not being pruned (known bug on some versions)used keeps climbing even as traffic drops; entries with expired timestamps lingershow table <name> twice: if the second call drops used, pruning was stuck. See diagnosis below

Quick checks

All of these are read-only runtime API calls against the stats socket.

# List all stick tables with capacity and current entry count
echo "show table" | socat unix-connect:/var/run/haproxy.sock stdio

The summary line per table includes size:<capacity> and used:<current-entries>. Compute the ratio for every table. Anything above 80% is a problem forming; 100% is a problem already happening.

# Inspect a specific table's entries (keys, counters, expiry)
echo "show table <table_name>" | socat unix-connect:/var/run/haproxy.sock stdio | head -50

# Find sources already over your intended rate threshold
echo "show table <table_name> data.http_req_rate gt 50" | socat unix-connect:/var/run/haproxy.sock stdio

# Confirm whether a traffic surge coincides with the fill
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(ConnRate|SessRate|CurrConns):"

# If peers are configured, check replication state
echo "show peers" | socat unix-connect:/var/run/haproxy.sock stdio

# Check whether denials are still firing as expected
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": dreq="$11" req_rate="$47}'

Note on that last check: the field positions assume the standard show stat CSV column order (dreq is column 11, req_rate is column 47 when splitting on commas). The expected symptom of overflow with nopurge is that dreq stops rising or drops relative to request rate, because new abusive clients are never tracked and therefore never denied. A healthy rate limiter under attack shows dreq climbing with the attack. A saturated table shows attack traffic rising while dreq stays flat.

How to diagnose it

  1. Get utilization for every table. Run show table and record used and size for each. Any table at 100% is overflowing right now. Any table above 80% needs sizing work before the next peak.

  2. Determine which overflow mode you are in. Check the table’s stick-table line in the config for nopurge. If present, overflow means new keys are refused: expect new clients to be untracked. If absent, overflow means silent eviction of expired entries: expect tracked clients to lose their counters and effectively reset.

  3. Check whether eviction is keeping up. Sample show table twice, a minute apart. If used stays pinned at size while you know new clients are arriving (frontend session rate is nonzero), entries are being created and evicted in a churn. Your rate window is effectively broken for anyone whose entry gets evicted before the ACL threshold trips.

  4. Look for the pruning bug. There is a known issue on HAProxy 2.7.x (and a related fixed one on 2.0.10-2.0.12, fixed in 2.0.13) where expired entries stop being pruned and used climbs monotonically. The signature: used keeps growing even as traffic falls, and running show table <name> itself forces eviction on the next call, dropping used. If your monitoring polls show table, it may be masking the bug by forcing cleanup on every poll.

  5. Correlate with attack signals. Compare frontend rate/req_rate trends against dreq. Under a real attack with a working rate limiter, dreq climbs. If req_rate from new sources is climbing and dreq is flat while the table sits at 100%, the limiter is defeated.

  6. Check the peers angle. If peers replication is configured, run show table on each peer. A full table on one node means all nodes are full. Fixing one node is not a fix.

  7. Estimate true cardinality. From show table <name>, sample the key space. For IP-keyed tables, count distinct entries and compare against your expected unique-client count per expire window. If the table holds entries for clients long gone, expire is too long. If it cannot hold one window’s worth of genuine clients, size is too small.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Stick table used / size ratio (via show table)The only direct overflow signal; absent from CSV stats> 80% sustained; 100% at any time on a security table
dreq rate vs frontend req_rateA working rate limiter under attack shows denials tracking the attackAttack traffic rising while dreq stays flat
Frontend ConnRate and SessRate (show info)Distinguishes “table full because of attack” from “table full because undersized”Spike in new connections from diverse sources coinciding with fill
New-entry creation rate vs expiry ratePredicts overflow before it happensCreation rate durably exceeds expiry rate at peak
show peers connection state and per-peer table statePeers replicate full state; a full table propagatesPeer disconnected, or peer tables also at capacity
Per-source http_req_rate / conn_rate (show table <name> data...)Confirms whether the limiter is actually seeing the abusersKnown-abusive sources absent from the table during an incident

Fixes

Undersized table

Increase size on the stick-table line. Memory cost is modest: roughly 80-100 bytes per entry for IPv4 keys with rate counters, and around 50 bytes base plus key length for string keys. A 1M-entry IPv4 table is on the order of tens of megabytes. Size for peak unique keys per expire window plus headroom, not for average. This requires a reload to take effect; if you use peers, table definitions must match across peers.

Tradeoff: larger tables cost memory and slightly more lookup work, but this is almost always the right fix when cardinality estimates were wrong.

expire too long

Shorten expire so stale entries leave before the table fills. The right value is tied to your rate-limit window: entries only need to live slightly longer than the longest window your ACLs evaluate (for example, http_req_rate(60s) needs entries to survive a bit over 60 seconds of inactivity, not an hour).

Tradeoff: too short an expire breaks session persistence tables, where clients legitimately go idle between requests. Size persistence tables for expire equal to your real session idle tolerance, and keep rate-limit tables on short expiry.

Attack-driven fill

If unique-source floods are the cause, sizing alone loses: an attacker controls cardinality and can fill any finite table. Shorten expire so entries churn quickly, consider tracking a coarser key (an IPv6 prefix rather than a full address, if you key on IPv6), and add upstream filtering so HAProxy is not the first line absorbing a cardinality attack. IPv6 diversity makes per-IP tracking weak in general; attackers can rotate through enormous address space.

nopurge behavior is wrong for your use case

If the table exists for rate limiting, silent refusal of new entries is usually the worse failure mode: new attackers are never tracked. Removing nopurge restores eviction-based behavior, which at least keeps the limiter working for tracked clients. If you genuinely need strict admission (no eviction ever), then monitoring the used/size ratio with an alert is not optional; it is the only way you will know the table stopped accepting entries.

Stuck pruning (version bug)

If you confirmed expired entries are not being pruned, upgrading past the affected version is the durable fix. As an operational workaround, periodic show table <name> calls force eviction, which is why some monitoring scripts accidentally mask the bug. Treat the workaround as temporary.

Prevention

  • Collect show table on a schedule. This data is not in show stat CSV output and not on the stats page. Scrape it via the runtime socket, record used and size per table, and graph the ratio.
  • Alert at 80% utilization. Page-worthy at 100% for any table serving rate limiting or security functions, ticket-worthy above 80% for everything else. The 80% line gives you time to resize before the limiter silently degrades.
  • Alert on the dreq divergence. During known attack traffic, a flat or falling dreq against rising req_rate is the behavioral signature of a defeated limiter. This catches overflow even if your show table collection lapses.
  • Size from measurement, not guesses. After a few weeks of production traffic, measure unique keys per expire window at peak and set size to at least 2x that. Revisit after traffic growth, launches, or marketing events.
  • Match expire to the enforcement window. Rate-limit tables: slightly longer than the longest rate window. Persistence tables: the real session idle tolerance. Do not copy one table’s expire to another.
  • Monitor all peers. Because peers replicate full table state, alert on the worst peer, not the average.
  • Rehearse the failure. Periodically fill a test table in staging and confirm which failure mode your config produces. Knowing in advance whether your limiter fails open (untracked new clients) or churns (evicted counters) changes your incident response.

How Netdata helps

  • Netdata collects the HAProxy runtime API signals that surround a stick-table overflow: frontend session and request rates, dreq, and per-frontend error counters, so you can see attack traffic rising while denials stay flat.
  • Correlating dreq against req_rate on the same dashboard turns the behavioral signature of a defeated rate limiter into something visible in seconds instead of a postmortem finding.
  • Because stick-table utilization is not in the CSV stats, pairing Netdata’s HAProxy charts with a scheduled show table scrape closes the instrumentation gap; Netdata’s alerting can then watch the utilization ratio and fire at your 80% threshold.
  • Per-second granularity on session-rate and connection-rate charts makes it easy to line up the moment a table filled with the traffic event that filled it, which is the difference between “undersized table” and “cardinality attack.”