HAProxy server-template and dynamic backends: monitoring service-discovery routing

With a static HAProxy config, the server list lives in the config file. Capacity review is config review: count the server lines and you know what the backend can absorb. With server-template, the server list lives in DNS. HAProxy pre-provisions a pool of empty server slots, and its internal resolver fills and empties them as A or SRV records change. This is the mechanism behind HAProxy fronting Kubernetes Ingress, Consul, and most autoscaled service-discovery setups.

That shift moves the failure domain. The resolver becomes a runtime dependency whose failure does not look like an outage: servers stay UP on cached IPs until those IPs stop working. The act count becomes a discovery signal, not just a health signal. And depending on how your control plane pushes changes, reloads can go from rare to constant, with real resource costs.

This article covers how the mechanism works, what changes operationally versus static backends, and which signals tell you the discovery pipeline is healthy.

What server-template is and why it matters

server-template declares a named pool of server slots in one line:

backend web
  balance roundrobin
  server-template web 10 _web._tcp.service.consul resolvers mydns check init-addr none resolve-opts allow-dup-ip resolve-prefer ipv4

The arguments are: a name prefix (web), a slot count (10), and a DNS name to resolve. For SRV records, the port comes from the record itself, so it is omitted from the template line. For plain A records, the port is given in the template and the record supplies only IPs. At runtime, HAProxy materializes slots as web1, web2, and so on, and the resolver section (resolvers mydns) keeps them populated.

Every property you took for granted from static config is now dynamic. Server identity, count, and addresses can all change without a config edit or a reload. Monitoring that assumes a fixed inventory will misread this environment.

How it works

flowchart LR
  DNS["DNS A or SRV records"] -->|queried on TTL cadence| RES["resolvers section"]
  RES -->|IP and port answers| POOL["server-template slot pool"]
  POOL -->|per-slot probing| HC["health check engine"]
  HC -->|UP / DOWN| ACT["backend act count"]
  ACT -->|serves traffic| APP["live endpoints"]
  RES -.->|failure| STALE["stale cached IPs"]
  STALE -.->|connect failures| ECON["econ rises"]

The lifecycle of a slot:

  1. Template expansion. At startup, HAProxy creates the full slot pool as named servers (web1 through web10). With init-addr none, HAProxy boots even if DNS is unreachable; slots start unresolved and are resolved at runtime. This is the standard pattern for container environments.
  2. Resolution. The resolver queries the configured nameservers on a TTL-driven cadence. SRV answers carry both IP and port; A answers carry IPs only.
  3. Slot assignment. Answers map into slots in order. As records appear and disappear, slots are reassigned. A slot’s IP can change over its lifetime, which has a monitoring consequence covered below.
  4. Health checking. The health check engine probes each populated slot independently of resolution. A resolved slot that fails checks leaves the active set until it recovers.
  5. Routing. The backend’s act count is the number of active-role slots currently UP. HAProxy load-balances only across those.

Two details drive most of the operational surprises. First, when DNS records are removed, some HAProxy versions do not immediately clear the stale address from the slot; the health check is the backstop that takes the dead endpoint out of rotation. Second, when the resolver itself fails, HAProxy keeps using its cached results until they expire, so a resolver outage surfaces later as connection errors to dead IPs, not as an immediate error anywhere.

Where dynamic backends show up

  • Kubernetes Ingress (HAProxy Ingress Controller). Endpoints churn with pod scheduling, rollouts, and autoscaling. Slot counts must cover peak replica counts, and endpoint churn during rollouts is constant background noise.
  • Consul service discovery. Services registered in Consul are consumed through Consul’s DNS interface, typically via SRV records.
  • Any autoscaled group behind DNS. Anything where the backend inventory changes faster than you want to edit config files.

In all of these, DNS TTL controls how quickly topology changes propagate into HAProxy. A long TTL means HAProxy reacts slowly to scale-down events; a short TTL increases query load on the resolver and the DNS server.

Operational differences from static config

The resolver is a single point of silent failure. In a static config, DNS is a startup-time concern. Here it is a runtime dependency. A resolver that stops answering does not down any server; HAProxy serves from cache until entries expire, then keeps routing to whatever it last knew. show resolvers (sent queries, received answers, timeouts, errors) is the only place this degradation surfaces before traffic breaks.

act is now a discovery metric. With static servers, a dropping act count means health failures. Here it can also mean DNS returned fewer endpoints, records were deleted, or slots are exhausted. Alert on act relative to the expected endpoint count for the service, not just on act = 0. Note that act and bck only count servers currently UP; for the full picture (DOWN, MAINT, DRAIN), count individual server rows by their status field.

Slots cap how many endpoints receive traffic. If DNS returns more endpoints than the template has slots, the extras get no traffic and nothing errors. Size the pool for peak replica count, and watch act approaching the slot count as a capacity signal on the discovery side.

A slot name is not a machine. web3 is a slot, not an identity. Its IP can change when records churn. Per-server time series will show a slot’s behavior jump when it remaps, and per-server econ can spike on a slot that just received a stale assignment. Interpret per-server metrics accordingly; the backend aggregate is usually the more honest signal.

Endpoint churn shows up as econ and check failures. When a pod is rescheduled, its old IP goes stale. Connections to the slot fail (refused or timed out), econ increments, and health checks fail with last_chk showing L4CON or L4TOUT until the slot re-resolves. Some churn-driven econ during rollouts is expected; baseline it so you can tell rollout noise from a resolver outage.

Reloads may be driven by discovery events. Some control planes react to endpoint changes by regenerating config and reloading HAProxy. Frequent reloads produce the classic reload storm: old processes lingering in soft-stop (Stopping: 1), file descriptor and memory pressure, and counters resetting on every cycle. Where possible, prefer resolver-driven updates or Runtime API server add/delete operations, which change the live server list without a reload. HAProxy 3.0 also added safer drain-before-delete primitives for this path. If reloads are unavoidable, set hard-stop-after so draining processes cannot linger indefinitely on long-lived connections.

Counter resets corrupt naive alerting. Every reload zeroes cumulative counters (econ, hrsp_5xx, wretr). Rate-based alerts that do not handle resets will see phantom drops and spikes. Use Uptime_sec to tag resets, as covered in the counter-reset guide linked below.

Per-slot health checks have a cost. Every populated (and provisioned) slot carries health check scheduling overhead. Community reports describe high CPU usage with very large slot counts (on the order of a thousand or more). Keep slot counts close to what the service actually needs.

A few config-level traps are specific to this pattern. On Kubernetes, multiple records may resolve to the same IP, which HAProxy rejects by default; resolve-opts allow-dup-ip fixes this, but there is a documented case where the option is silently reset when placed on a default-server line while the template uses resolvers, so put it on the server-template line itself. HAProxy prefers IPv6 answers when the host has an IPv6 stack; set resolve-prefer ipv4 explicitly if IPv6 is not actually routable in your environment. Finally, after a reload that restores server state from a state file, some versions do not re-resolve SRV names promptly, leaving slots with stale addresses. After any reload, verify that act returns to the expected count.

Reading the live state

These checks are safe and read-only:

# Live server list, including auto-generated slot names, addresses, and state
echo "show servers state" | socat unix-connect:/var/run/haproxy.sock stdio

# Active/backup counts per backend
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": active="$20" backup="$21}'

# Resolver health: queries sent, answers received, timeouts, errors
echo "show resolvers" | socat unix-connect:/var/run/haproxy.sock stdio

# Reload and resolution-failure indicators
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(Uptime_sec|Stopping|FailedResolutions):"

# Connection errors per backend and server (stale-IP detection)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": econ="$14}'

# How many HAProxy processes are running (reload storm check)
pgrep -c haproxy

show servers state is the ground truth for “what does HAProxy think the service looks like right now.” Compare its addresses against a direct DNS lookup of the same name when you suspect staleness.

Signals to watch in production

SignalWhy it mattersWarning sign
show resolvers timeouts and errorsOnly direct view of resolver health before routing breaksAnswers lagging queries; timeout or error counts growing
FailedResolutions (show info)Cumulative resolution failuresAny sustained increments; should sit at zero in steady state
Backend act countMirrors discovered, healthy endpointsact below expected endpoint count, or dropping without a deploy
Per-server status and last_chkWhich slots fail and why (L4TOUT/L4CON point at dead IPs)Multiple slots failing with L4TOUT/L4CON after endpoint churn
econ rate per serverConnections attempted to stale or dead IPsecon climbing on slots whose addresses DNS no longer returns
wretr / wredisChurn being masked by retries and redispatchesRetry rate rising during rollouts and staying elevated afterward
Uptime_sec and HAProxy PID countReload frequency and reload-storm detectionUptime repeatedly resetting; more than 2-3 concurrent HAProxy processes
Stopping durationDrain behavior after reloadsSoft-stop lingering beyond roughly 10 minutes
act vs slot countDiscovery-side capacity ceilingact pinned at the template slot count while the service scales past it

The correlation that shortens most incidents in this architecture: resolver errors rising first, then econ, then servers flapping DOWN with L4TOUT. That ordering says “DNS broke, and now the cached topology is decaying,” which sends you to the resolver and the DNS server rather than to the backends.

How Netdata helps

  • Netdata’s HAProxy collector reads the stats CSV over the runtime socket or HTTP stats endpoint, so per-backend act counts, per-server status, econ/eresp, and retry counters are charted continuously rather than sampled by hand during incidents.
  • Historical act per backend makes “DNS started returning fewer endpoints” visible as a step change, distinguishable from gradual health-check-driven loss.
  • Per-server econ correlated with server status transitions pinpoints stale-IP churn during rollouts and separates it from application-level failures.
  • Reset-aware handling of cumulative counters (with Uptime_sec as the reset marker) keeps frequent reloads from rendering as phantom drops in error rates.
  • Comparing frontend versus backend 5xx rates alongside econ distinguishes HAProxy-generated 503s (no slots left UP) from errors the discovered endpoints returned themselves.