HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable

You look at the HAProxy stats page (or your monitoring) and a server row is red. The last_chk field says L4TOUT or L4CON, status says DOWN, and HAProxy has stopped routing traffic to that server. Both codes mean the failure happened before any HTTP, before any application code, at the plain TCP layer.

L4TOUT means HAProxy opened a socket and the TCP connect did not complete within the check timeout. L4CON means the connect attempt failed immediately with an error, typically “Connection refused” (TCP RST) or “No route to host” (ICMP). The server is unreachable at the TCP layer, and after fall consecutive failures (default 3), HAProxy marks it DOWN and redistributes its traffic.

What this means

HAProxy’s health check engine runs independently of traffic. On every inter interval (default 2000ms), it opens a fresh TCP connection to each server’s check port. What happens next determines the code:

  • L4CON: the connect call failed right away. Something actively rejected the connection: the far end sent RST (nothing listening on that port), a firewall sent an ICMP error, or a middlebox reset the connection. The key property: the answer came back fast.
  • L4TOUT: the connect call never completed within the timeout. SYN went out, no SYN-ACK came back. The key property: silence. This is packet loss, a dropping firewall, an overloaded host that cannot answer SYNs, or a wrong/unroutable address.

That distinction, refusal versus silence, is the first fork in the diagnosis and it shows up directly in the check duration. An L4CON check typically completes in about 0ms. An L4TOUT check burns the entire timeout, for example around 2001ms when the timeout equals the default 2000ms inter. The check_duration field in the stats tells you which one you have before you touch anything else.

One subtlety on timeouts: if you do not set timeout check explicitly, HAProxy uses inter as the complete check timeout. With the default inter of 2000ms, a backend that is merely slow to accept connections for two seconds flaps to L4TOUT. If you do set timeout check, HAProxy uses min(timeout connect, inter) as the connect timeout and timeout check as an additional read timeout. Know which regime your config is in before you conclude the network is broken.

The state machine looks like this:

flowchart TD
    A[Check fires every inter] --> B{TCP connect result}
    B -->|SYN-ACK, connect OK| C[L4OK - server stays UP]
    B -->|RST or ICMP error| D[L4CON - refusal]
    B -->|no reply within timeout| E[L4TOUT - silence]
    D --> F[chkfail +1]
    E --> F
    F --> G{failures >= fall?}
    G -->|no| H[Server UP, transitional state]
    G -->|yes| I[Server marked DOWN, chkdown +1]
    I --> J[Traffic redistributed to survivors]

While failures accumulate below fall, the server stays UP and keeps receiving traffic. In stats output you may see the code with an asterisk prefix (for example *L4CON), indicating a transitional state: HAProxy has seen the failure but has not yet hit the fall count. Once chkdown increments, the server is DOWN and out of rotation until rise (default 2) consecutive checks pass again.

Common causes

CauseWhat it looks likeFirst thing to check
Backend process down or crashedL4CON, check duration ~0ms, single server affectedIs the service listening? ss -tlnp on the backend
Wrong check port or firewall blocking HAProxy’s source IPL4CON, but nmap from your workstation shows the port openTest connect from the HAProxy host itself, not from elsewhere
Dropping firewall or security groupL4TOUT, check duration equals the timeout, one server or one subnetPacket capture on the HAProxy host: do SYNs leave, do SYN-ACKs return?
Backend overloaded, SYN backlog full or CPU starvedL4TOUT, possibly flapping, high ctime on surviving trafficBackend CPU, listen queue overflow counters on the backend
Network partition or routing changeMultiple servers L4TOUT at once, same lastchg windowCount how many servers changed state simultaneously
HAProxy-side resource starvationMany backends L4TOUT at once under heavy load, backends prove healthyHAProxy Idle_pct, scur/slim, FD headroom
Missing timeout check with slow backendsSpurious L4TOUT at ~2000ms duration, server otherwise healthyCompare check_duration to your effective check timeout
Stale DNS (server-template deployments)L4CON or L4TOUT after instance replacement, IP no longer existsshow servers state IPs versus current DNS records

The single most useful heuristic: if many servers across different backends show L4TOUT/L4CON at the same time, suspect the network or HAProxy itself, not the applications. Applications fail one at a time; networks and proxies fail in batches.

Quick checks

All read-only. Run them on the HAProxy host.

# 1. Server status, last check result, and failure counters
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" chkfail="$22" chkdown="$23" lastchg="$24"s last_chk="$37}'

# 2. Check duration: ~0ms means refusal (L4CON), ~timeout means silence (L4TOUT)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": check_duration="$39"ms"}'

# 3. What address and port HAProxy is actually checking
echo "show servers state" | socat unix-connect:/var/run/haproxy.sock stdio

# 4. Reproduce the check from the HAProxy host (replace IP/port with step 3 output)
time curl -v --max-time 3 telnet://10.0.0.12:8080 </dev/null

# 5. Is HAProxy itself starved? Low Idle_pct means the event loop is struggling
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(Idle_pct|CurrConns|Maxconn):"

# 6. Connection errors on real traffic (econ), not just checks
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": econ="$14" ctime="$60"ms"}'

Notes on interpretation:

  • Step 4 matters more than it looks. A classic false lead is running nmap or curl from your laptop, seeing the port open, and concluding HAProxy is wrong. The health check originates from the HAProxy host with its source IP. Host firewalls, security groups, and network ACLs routinely allow your admin range while dropping the proxy’s. Always reproduce from the HAProxy host.
  • If step 1 shows several servers with nearly identical small lastchg values, they all failed in the same window. That is a network or shared-infrastructure signature.
  • Step 5 catches the under-load case: when HAProxy is connection- or CPU-saturated, check sockets compete with proxied traffic and checks time out even though the backends are fine.

How to diagnose it

  1. Read the code and the duration together. last_chk gives you L4CON versus L4TOUT; check_duration confirms it. L4CON at ~0ms is a refusal. L4TOUT at ~the check timeout is silence. Everything downstream depends on which one you have.

  2. Determine the blast radius. One server or many? One backend or several? Use the lastchg timestamps. Simultaneous failures point at the network path, a shared dependency, or HAProxy. A single server points at that server or its specific path.

  3. Verify what HAProxy is dialing. show servers state shows the exact address and port of the check. In server-template/DNS setups, confirm that IP still exists. A stale resolution after a container or instance replacement produces L4CON (nothing listening on the old IP) or L4TOUT (the old IP is now unrouteable). Cross-check with show resolvers if you use DNS discovery.

  4. Reproduce from the HAProxy host. TCP-connect to the check address from the proxy itself (step 4 above). Refused: nothing is listening there, or a firewall on the backend rejects HAProxy’s source. Timeout: packets are being dropped between the two, or the backend cannot answer SYNs.

  5. For L4TOUT, confirm with a capture. On the HAProxy host, capture traffic to the server IP. If SYNs leave and no SYN-ACK returns, the loss is on the path or the far end. If SYNs never leave, the problem is local to HAProxy (resource starvation, ephemeral port pressure).

  6. Rule out HAProxy-side starvation. Check Idle_pct (below ~20% means the event loop is under stress; below ~5% it is saturated), scur/slim ratios, and FD headroom. Under heavy load, health check connections get starved just like everything else and healthy backends flap to L4TOUT. Also check ctime and econ on real traffic: if traffic connections to the same server succeed while checks time out, the server is fine and the proxy is the bottleneck.

  7. Check the backend’s accept path. For a refused or silently dropping backend, look at the backend host: is the process listening, is its CPU pegged, is its kernel SYN/accept queue overflowing? An overloaded backend that cannot dequeue accepts produces L4TOUT, not L4CON, which routinely gets misread as a network problem.

  8. Check your timeout budget. If you have not set timeout check, your effective total check timeout is inter (2000ms by default). If check_duration hovers near that value even on successful checks, you are one slow GC pause away from spurious L4TOUT. That is a configuration issue, not an outage.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
last_chk per serverThe exact failure code, your first classificationL4TOUT or L4CON replacing L4OK/L7OK
check_durationDistinguishes refusal from timeout at a glanceDuration pinned at the check timeout
chkfail rateFailures accumulating, possibly below the fall thresholdRapid increments on a server still showing UP
chkdownConfirmed DOWN transitionsIncreasing on multiple servers at once
lastchgSeconds since last state change; clusters reveal correlated failureMany servers with similar small values
econ per serverReal-traffic connection errors, the traffic-side mirror of L4CON/L4TOUTSustained nonzero rate on the same server the checks flag
ctime per serverConnect latency on real traffic; separates slow-path from dead-pathHigh ctime paired with econ (timeouts) versus zero ctime with econ (refusals)
act per backendHow many servers remain in rotationBelow 50% of the pool, cascade risk
Idle_pctWhether HAProxy itself is too busy to run checksBelow 20% during an L4TOUT flap storm

Correlating last_chk with econ and ctime is the fastest way to tell “health check says dead but traffic flows fine” (proxy-side or check-config problem) from “health check and traffic both fail” (server or network problem).

Fixes

Backend process down (L4CON, single server)

Restart or repair the service on the backend. HAProxy returns the server to rotation automatically after rise consecutive checks pass; you do not need to touch HAProxy. If the server keeps crashing, put it in MAINT via the runtime API to stop the flap while you investigate, rather than letting it oscillate and repeatedly shift traffic.

Firewall or check-port mismatch (L4CON or L4TOUT, reproducible from HAProxy host)

Fix the rule or the port. Verify HAProxy is checking the port you think it is: if you do not specify a separate check port, the check goes to the server’s service port. If the service port is intentionally firewalled to specific sources, allow the HAProxy hosts explicitly. Re-test with a connect from the HAProxy host before declaring victory.

Network partition or routing issue (L4TOUT, multiple servers)

This is not a HAProxy fix. Fail traffic away from the affected path or AZ if you have that capability. HAProxy is already doing the right thing by draining the unreachable servers; watch act and survivor scur/slim so the redistribution does not overload the remaining servers into a collapse cascade.

HAProxy-side starvation (L4TOUT under load, backends healthy)

Relieve the proxy: raise capacity, shed load, fix whatever is consuming the event loop (TLS handshake storms and maxconn saturation are the usual suspects). If scur is near slim, you are connection-saturated; see the related guide on scur approaching slim. Do not “fix” this by lengthening check timeouts alone, since that only delays detection while the proxy remains overloaded.

Spurious L4TOUT from a missing timeout check

Set timeout check explicitly for backends whose health endpoint or accept path is non-trivial (databases, heavyweight apps). Relying on the default behavior ties your detection timeout to inter, and 2000ms is aggressive for anything that does real work in the check. Tradeoff: a longer check timeout slows failure detection by up to that amount per check, multiplied by fall.

Stale DNS in server-template deployments (L4CON/L4TOUT after instance replacement)

Fix the resolver path: check show resolvers for timeouts and errors, verify the DNS service is healthy, and confirm TTLs are sane. As a temporary measure you can force server addresses via the runtime API. Treat recurring cases as a resolver-availability problem, not a server problem.

Prevention

  • Set timeout check explicitly on every backend. Know your effective connect timeout (min(timeout connect, inter) when timeout check is set, inter alone otherwise) instead of inheriting a default you have never thought about.
  • Tune fall and inter deliberately. The defaults (fall=3, inter=2s) mean roughly 6 seconds to mark a server DOWN. Faster detection sends more traffic to a dying server per failure window; slower detection tolerates more flapping. Pick per backend criticality.
  • Alert on chkfail rate, not just status. A server accumulating check failures while still UP is your early warning; waiting for DOWN wastes the fall window.
  • Alert on correlated lastchg. Multiple servers changing state in the same minute is a network event and should page differently than a single server dying.
  • Monitor HAProxy’s own headroom (Idle_pct, scur/slim, FD usage) so you can tell proxy-side check starvation from real backend failure without guesswork.
  • Test checks from the proxy host in CI or deploy validation. A firewall change that blocks the health check path shows up as a production incident otherwise.

How Netdata helps

  • Per-server last_chk, chkfail, chkdown, and lastchg collected every second, so you see failures accumulating below the fall threshold and catch the exact moment a server transitions, not the next stats-page refresh.
  • check_duration alongside the failure code, making the refusal-versus-timeout distinction visible on a chart instead of requiring manual CSV parsing during an incident.
  • Correlation of check failures with econ and ctime on live traffic, which separates “checks fail but traffic flows” (proxy-side) from “both fail” (server or network) in one view.
  • Fleet-wide view across backends, so simultaneous L4TOUT/L4CON across servers shows up as a correlated event, pointing at network or HAProxy load rather than application failure.
  • HAProxy self-headroom signals (Idle_pct, scur/slim) on the same dashboard, so ruling out proxy-side check starvation takes seconds.