Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion

pulsar_active_connections has been drifting upward for weeks. Not spiking, not crashing, just rising a few connections a day. Then one morning new producers start failing to connect, or the broker dies with OutOfDirectMemoryError, and the postmortem shows the leak was visible the whole time.

Every TCP connection costs the broker one file descriptor and a slice of Netty direct memory. Connections that are established but never closed accumulate silently until one of those two resources hits its ceiling, at which point the failure is abrupt: refused connections or a crash, with no graceful degradation in between.

The distinguishing feature of this failure mode is the timescale. Instantaneous connection counts mean almost nothing. The signal is the multi-week trend and, more precisely, the accounting: connections created minus connections closed should equal active connections. When that arithmetic stops balancing, something is leaking.

What this means

A broker’s active connection count is bounded by two hard limits, and either one can kill it:

• File descriptors. Each connection consumes one FD. When the process hits its ulimit -n ceiling, new connections fail immediately, with no warning period. A production broker should run with a 100K+ FD limit; a broker running with a systemd default in the low thousands has almost no headroom for even modest leaks.
• Netty direct memory. Each connection’s I/O buffers live off-heap. Direct memory exhaustion is invisible to heap monitoring and ends with io.netty.util.internal.OutOfDirectMemoryError or a broker that hangs waiting for buffer allocation.

The leak usually sits on the client side: an application that constructs new client or producer instances per request or per pod without closing them, or a retry loop that opens connections faster than the broker can age them out. Less commonly it sits in the broker or proxy itself, in which case the broker half of the socket lingers in CLOSE_WAIT after the client has gone away.

There is also a variant that looks like this problem but is not: connection count stable while direct memory keeps growing. That is a Netty buffer leak, not a connection leak. Connections are being closed correctly, but their buffers are not being released. The diagnosis path is different, so separate the two early.

flowchart TD
  A[Active connections climbing over weeks] --> B{created minus closed equals active?}
  B -- "yes, balanced" --> C[Real load growth or client-side connection multiplication]
  B -- "created grows, closed does not" --> D{Which resource is filling?}
  D -- "FDs near ulimit" --> E[Connection leak: audit clients, check CLOSE_WAIT]
  D -- "Direct memory near MaxDirectMemorySize" --> F[Per-connection buffer accumulation]
  B -- "active stable, direct memory growing" --> G[Netty buffer leak: enable leak detection]
  E --> H[Fix client lifecycle, raise FD limit as buffer]
  F --> H
  G --> I[Capture leak report, restart, upgrade]

Common causes

Quick checks

# Current active connections and churn counters
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_active_connections|pulsar_connection_(created|closed)_total_count"

# Check the accounting: created minus closed should equal active.
# Do the subtraction across several samples over hours or days, not one scrape.

# Broker's actual FD usage and configured limit
BROKER_PID=$(pgrep -f PulsarBroker | head -1)
ls /proc/$BROKER_PID/fd | wc -l
grep "Max open files" /proc/$BROKER_PID/limits

# Direct memory pool (Pulsar does not expose this via Prometheus).
# JMX java.nio:type=BufferPool,name=direct is the reliable source if you have an exporter.
# VM.native_memory only works if NativeMemoryTracking was enabled at JVM start.
jcmd $BROKER_PID VM.native_memory summary 2>/dev/null || true

# Half-closed sockets piling up on the broker
ss -tan state close-wait | wc -l
ss -tan state time-wait | wc -l

# Throttling already kicking in (broker protecting itself)
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_throttled_connections

# Netty allocator state (if the process is still responsive)
curl -s http://<broker-host>:8080/admin/v2/broker-stats/allocator-stats/default

# Log evidence of either ceiling
grep -iE "Too many open files|OutOfDirectMemoryError|Direct buffer memory" /var/log/pulsar/*.log | tail -20

All of these are read-only. The allocator-stats endpoint can be slow on a heavily loaded broker; treat it as lower priority during an active incident.

How to diagnose it

1. Establish the trend, not the value. Pull pulsar_active_connections over 2-4 weeks. A smooth upward ramp points to a leak. A flat line with sharp steps points to reconnection events after bundle unloads or restarts. Random sawtooth behavior with recovery is normal client churn.

2. Do the arithmetic. Sample pulsar_connection_created_total_count and pulsar_connection_closed_total_count at two points in time. The delta of created minus the delta of closed should approximately equal the change in active connections. If created outpaces closed persistently and active climbs in lockstep, clients are opening connections they never close. If created and closed both move but active still climbs, the broker is not reaping its side.

3. Identify which ceiling you are approaching. Compare ls /proc/<pid>/fd | wc -l against Max open files in /proc/<pid>/limits. Separately, check direct memory usage against -XX:MaxDirectMemorySize (broker default 4GB). Whichever ratio is higher tells you how much time you have and which failure you’ll get.

4. Rule out the buffer-leak variant. If active connections are flat but direct memory is growing, stop chasing clients. Connections are closing correctly but their buffers are never released. Check Netty allocator stats via the broker-stats endpoint and prepare to capture a leak report (see Fixes).

5. Localize the leak. A large CLOSE_WAIT count on the broker means the remote end closed and the local process never called close on its socket, so the leak is broker/proxy-side or in whatever the broker is talking to. A large count of established connections from one client IP range points at a specific application. ss -tanp shows owning processes for broker-local sockets.

6. Correlate with cluster events. Overlay the connection curve with pulsar_lb_unload_bundle_total and broker restart timestamps. Step increases that line up with unloads are client reconnection storms, and the fix is in load-balancer behavior or client reconnection config, not leak hunting.

Metrics and signals to monitor

Fixes

Client-side connection leak

The most common root cause is application code that creates a new PulsarClient, producer, or consumer per operation and never closes it. Pulsar clients are expensive and designed to be long-lived and shared.

• Audit the application for client instance lifecycle. One shared client per process, closed on shutdown, is the correct pattern. Per-request or per-message client construction is a leak by design.
• Check whether a client library version or wrapper (framework integration, sidecar, or script) spawns connections per task.
• As a short-term pressure valve, restart the offending application instances. This drops their connections and buys time, but the leak will return.

Reconnection storms after bundle unloads

If the connection curve steps up after bundle unload events, clients are reconnecting and old broker-side connections are not draining cleanly. Pulsar clients reconnect automatically, so the connections themselves are expected; the question is why the old ones linger. Verify the created/closed arithmetic around an unload event, and review load balancer aggressiveness if unloads are frequent enough to keep the count elevated.

Broker-side half-closed sockets (CLOSE_WAIT pileup)

If the broker accumulates CLOSE_WAIT sockets, it is not closing its end of connections the peer has already closed. There is no application-level fix you can apply at runtime.

• Raise the FD limit immediately if you have not already: set LimitNOFILE=100000 or higher in the broker’s systemd unit (or the equivalent for your supervisor) and restart during a maintenance window. This converts an imminent outage into headroom.
• Tune kernel TCP keepalive so the kernel can detect and reap half-open connections. Pulsar relies on an application-level keepalive in its binary protocol, which does not detect every half-closed TCP state on its own. As a concrete starting point, core maintainer guidance in apache/pulsar#14826 suggests net.ipv4.tcp_keepalive_time = 1200, net.ipv4.tcp_keepalive_intvl = 60, net.ipv4.tcp_keepalive_probes = 20, which reduces half-close linger from the default two hours to roughly 20 minutes.
• Check your version against known leak fixes. Proxy connection leaks fixed in 2.8.2/2.9.0 (PR #11848) and client-side idle connection release from PIP-165 (idle connections not referenced by any producer, consumer, or transaction are closed automatically, controlled by connectionMaxIdleSeconds and connectionIdleDetectionIntervalSeconds) both address real leak paths. If you are on an affected older version, upgrading is the actual fix.

FD exhaustion in progress

If the broker is already logging “Too many open files”, you cannot raise the soft limit on a running process from outside. Your options are:

1. Shed connections at the client side (scale down or restart the leaking application) to free FDs without a broker restart.
2. Restart the broker, knowing it triggers a bundle redistribution and client reconnections across the cluster. Raise LimitNOFILE in the unit file first so the restart actually fixes the ceiling.

Netty buffer leak (stable connections, growing direct memory)

This is the distinct variant. Connections are fine; buffers are not being released.

• Enable Netty leak detection to capture evidence: -Dpulsar.allocator.leak_detection=Simple -Dio.netty.leakDetectionLevel=simple is considered production-safe; Advanced gives stack traces at a performance cost and is better suited to a canary broker. Leaks surface in logs as LEAK: ByteBuf.release() was not called reports.
• A broker restart is the only immediate relief once direct memory is near the ceiling. The broker is functionally dead at exhaustion, so treat this as a planned restart, not an optional one.
• Match the leak report against known issues for your Pulsar and BookKeeper versions. Direct memory regressions have historically been fixed by upgrades (for example, a BookKeeper direct memory climb on 2.10.1 was resolved in 2.10.4 via a BookKeeper upgrade). This class of bug is almost never fixed by tuning; it is fixed by patching. See Apache Pulsar OutOfDirectMemoryError for the crash-side analysis.

Prevention

• Trend, don’t threshold. Alert on the multi-week slope of pulsar_active_connections, not an instantaneous value. A connection count of 8,000 means nothing; a count that was 4,000 last month and 8,000 now, with flat traffic, means everything.
• Alert on the accounting gap. The difference between created and closed counters should track active connections. A persistent, widening gap is a leak alarm with weeks of runway.
• Set the FD limit deliberately. 100K+ on every production broker, enforced through the systemd unit or container runtime config, verified in /proc/<pid>/limits after every deploy. Do not trust OS defaults.
• Monitor FD ratio and direct memory ratio as first-class signals. The degradation curve is a cliff at 100%, so the ratio at 75% is the actionable one.
• Keep clients patched. Idle connection release and the proxy leak fixes exist because these leaks were common. Staying several minor versions behind keeps known leaks open.
• Load-test client lifecycle. Before rolling out a new application or client library version, watch the broker’s created/closed counters through a deploy and a few restart cycles. A client that leaks in staging leaks in production at scale.

How Netdata helps

• Long-retention trending of pulsar_active_connections. Leaks live on the multi-week timescale that short-retention monitoring erases. Per-second collection with long retention makes the slow ramp visible and lets you compare against traffic baselines.
• Churn counter correlation. Netdata charts pulsar_connection_created_total_count and pulsar_connection_closed_total_count alongside active connections, so the created-minus-closed accounting check is a glance at one dashboard instead of a manual subtraction exercise.
• FD usage vs limit per process. Process-level file descriptor counts charted against the configured limit turn an invisible cliff into a percentage gauge with a trend line.
• Process memory decomposition. RSS vs heap visibility helps spot the direct-memory variant, where the process balloons while JVM heap dashboards stay green.
• Cross-signal context. Throttled connections, bundle unload rate, and broker publish latency on the same view lets you tie connection steps to cluster events and rule out reconnection storms in minutes.

Related guides

• Apache Pulsar broker down: telling a dead broker from a fenced one
• How Apache Pulsar actually works in production: a mental model for operators
• Apache Pulsar monitoring checklist: the signals every production cluster needs
• Apache Pulsar monitoring maturity model: from survival to expert
• Apache Pulsar OutOfDirectMemoryError: the off-heap crash JVM heap dashboards never show