ActiveMQ transport connector not accepting: a live broker that refuses one protocol

The broker process is running. The JVM is up, OpenWire clients on 61616 are producing and consuming, and your process-level health check is green. But the AMQP clients on 5672 cannot connect, and their reconnect loops are filling your application logs.

This partial failure is worse than a full outage in one specific way: most monitoring does not see it. A TCP check against one port, or a process-alive check, tells you nothing about the other four connectors. ActiveMQ Classic runs each transport connector (OpenWire 61616, AMQP 5672, STOMP 61613, MQTT 1883, WebSocket 61614) as its own accept path, and each one can fail independently while the rest of the broker looks healthy.

This article covers how to confirm which connector is refusing connections, the causes that account for nearly all of these incidents, and how to fix them without bouncing the broker as a first move.

What this means

A transport connector “not accepting” is one of three distinct conditions, and the fix differs for each:

1. The port is not listening at all. The connector never started, died, or was never configured. ss -tlnp shows nothing on the port.
2. The port is listening but connections are refused or reset. The accept path is blocked: file descriptor exhaustion, the connector’s maximum connection limit, or a hung accept thread.
3. The TCP connection completes but the protocol handshake fails. TLS certificate problems, authentication failures, or a protocol mismatch (plaintext client against an SSL connector). From the client side this looks identical to “not accepting,” but the broker is actively rejecting after accept.

Condition 3 wastes the most time, because nc -z succeeds and operators conclude the connector is fine. A listening socket only proves the JVM bound the port, not that it can complete a handshake or register a connection.

Two deployment contexts change what “not accepting” means before you start diagnosing:

• Shared-storage HA standby. The standby broker’s process runs but its transport connectors are not started while it waits for the store lock. A standby with no listeners is expected. Do not troubleshoot it.
• KahaDB recovery after an unclean shutdown. The broker may open ports before it is ready to serve clients during journal replay and index rebuild. If the broker restarted minutes ago after a crash, check whether recovery is still in progress before assuming a connector failure.

Common causes

Quick checks

All read-only. Run them against the affected connector’s port.

# 1. Is the connector actually listening, and which process owns it?
ss -tlnp | grep -E '61616|5672|61613|1883|61614'

# 2. Count established connections per connector port (OS-level view)
for p in 61616 5672 61613 1883 61614; do
  echo -n "port $p: "; ss -tn state established "( sport = :$p )" | wc -l
done

# 3. Does a raw TCP connect succeed? (passes even when the handshake will fail)
nc -z -w 3 localhost 5672 && echo "TCP OK" || echo "TCP REFUSED"

# 4. Recent accept failures in the broker log
grep -i "Could not accept connection" /opt/activemq/data/activemq.log | tail -30

# 5. FD exhaustion signature
grep -i "Too many open files" /opt/activemq/data/activemq.log | tail -10

# 6. Open FDs vs limit for the broker process
BROKER_PID=$(pgrep -f activemq | head -1)
echo "open: $(ls /proc/$BROKER_PID/fd | wc -l)"
grep 'Max open files' /proc/$BROKER_PID/limits

# 7. Sockets stuck in CLOSE_WAIT on the affected port (FD leak pattern)
ss -tn state close-wait "( sport = :5672 )" | wc -l

For the JMX view, Jolokia on the web console (port 8161, bound to 127.0.0.1 by default since 5.16) is the fastest path:

# Current connections across all connectors
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/CurrentConnectionsCount'

# Per-connector view (connectorName matches the name attribute in activemq.xml)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Connector,connectorName=amqp,brokerName=localhost'

If the failing connector is SSL, check the certificate directly. A completed TLS handshake against the port proves both the listener and the cert chain:

# Cert chain and expiry dates as presented by the broker
openssl s_client -connect localhost:61616 -servername localhost </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject

How to diagnose it

Work the conditions in order. Most incidents resolve at step 2 or 3.

1. Classify the failure. Run checks 1-3 above. No listener: go to step 5. TCP refused or reset: steps 2-4. TCP connects but clients still fail: this is a handshake problem, go to step 4.

2. Check file descriptors. Compare open FD count against the process limit (check 6). The classic production misconfiguration is the default ulimit of 1024, which a broker with a few hundred connections plus KahaDB journal files and log files will exhaust. Also count CLOSE_WAIT sockets on the affected port (check 7): a large and growing CLOSE_WAIT count means the broker accepted sockets but never closed them, leaking FDs even though the live connection count looks normal.

3. Check the connector’s own limit and the log. Look for ExceededMaximumConnectionsException in the broker log. The TCP transport has a maximumConnections option (default effectively unbounded), but if someone set it, the broker refuses new connections past the limit on that connector only. Advisory topics and connection pools can consume more connections than you expect, so verify the actual count against the limit before concluding the limit is wrong.

4. Distinguish TLS, protocol, and auth failures. All three happen after TCP accept, so nc -z is useless here. In the broker log: SSLHandshakeException points at certificate or cipher problems (check expiry with the openssl command above); Unrecognized SSL message, plaintext connection? means a plaintext client is hitting an SSL connector, which is a client configuration bug, not a broker bug; repeated authentication failed lines mean the connector is fine and the credential path is broken. The tell for auth: the failure affects every connector that uses the same security plugin, not just one port.

5. If the port never listened, read the startup log. A connector that fails to bind (port already in use, bad URI, missing keystore for an SSL scheme) logs an error at startup and the broker continues without it. ss -tlnp in check 1 shows you whether something else owns the port.

6. If nothing above explains it, take a thread dump. Especially on nio:// connectors, where a shared selector pool replaces the one-thread-per-connection model: a stuck selector or a hung handshake in the NIO path can stop accepts on one connector while plain TCP connectors on the same broker keep working. There are known issues with the NIO+SSL transport hanging during or after the handshake. Capture two thread dumps 10 seconds apart and look for transport threads in the same state in both:

# Capture thread dumps for offline analysis (low impact, but briefly stalls the JVM)
jstack $BROKER_PID > /tmp/amq-tdump-1.txt
sleep 10
jstack $BROKER_PID > /tmp/amq-tdump-2.txt

7. Rule out the GC pause masquerade. A long full GC freezes all connectors simultaneously, clients hit wireFormat.maxInactivityDuration (default 30000ms) and disconnect, and the post-pause reconnect burst can look like a connector failure. If all connectors stalled together and recovered together, check GC pause duration before blaming any single connector. That is a different incident; see the related GC guide below.

flowchart TD
  A[Connector not accepting] --> B{Port listening?}
  B -- No --> C[Startup bind or config failure - read broker log from startup]
  B -- Yes --> D{TCP connect succeeds?}
  D -- No --> E{FDs at limit?}
  E -- Yes --> F[FD exhaustion - check CLOSE_WAIT leak and ulimit]
  E -- No --> G[Connector connection limit or hung accept path - thread dump, check NIO]
  D -- Yes --> H{Handshake completes?}
  H -- No --> I[TLS cert, protocol mismatch, or auth backend - read log error type]
  H -- Yes --> J[Connector healthy - investigate client or network path]

Metrics and signals to monitor

Fixes

File descriptor exhaustion

Raise the process limit to at least 65536 and restart the broker during a window; the limit cannot be raised for a running JVM from outside. Before restarting, check whether the FD growth is a leak rather than legitimate connections: a rising CLOSE_WAIT count on one port means accepted sockets are not being closed, and raising the limit only buys time. Track down the client or network device (load balancers that drop idle connections are a common source) that leaves sockets half-closed. If the accept queue handoff is implicated, the useQueueForAccept transport option set to false has been reported as a workaround; treat this as version-specific and test it.

Connector connection limit

If maximumConnections is set on the connector URI and the broker is logging ExceededMaximumConnectionsException, either raise the limit or find out what is consuming connections. Look for advisory-topic consumers, leaked connections from applications that never close them, and connection pools sized larger than expected. Do not just raise the limit until you know the count is legitimate: a connection leak will consume any limit you set, and each connection costs an FD and, on the default TCP transport, a thread.

TLS certificate failure

Replace or renew the certificate in the broker’s keystore and restart the connector (in practice, this usually means a broker restart, so plan it). Then fix the process gap: certificate expiry is a calendar event and should never be discovered by a connector refusing handshakes. Alert on days-to-expiry well inside your rotation window.

Protocol mismatch

The log line Unrecognized SSL message, plaintext connection? is a client connecting without TLS to an ssl:// connector. Fix the client URI scheme. Nothing is wrong with the broker, and no broker change will help.

Auth backend failure

If every connector that shares a security plugin is rejecting after connect, the problem is the credential path: rotated passwords that clients have not picked up, or a dead LDAP/JAAS backend. Restore the backend or roll back the credential change. A single misconfigured client with aggressive reconnect and no backoff can also hammer a healthy connector hard enough to look like an outage; the log will show one source IP repeating.

Hung NIO accept path

If a thread dump shows the NIO selector or handshake path stuck, particularly with nio+ssl://, the only reliable immediate recovery is a broker restart during a maintenance window. Longer term, evaluate whether plain TCP transport or a patched broker version avoids the hang, and keep heap and GC healthy so selector threads are not starved.

Prevention

• Monitor per-connector, not per-process. The playbook threshold is direct: zero accepts for more than 30 seconds on a connector that should be serving, while the broker is up, warrants investigation. Count established connections per port or poll per-connector JMX MBeans; a single port check on 61616 tells you nothing about 5672, 61613, 1883, or 61614.
• Raise the FD limit at provisioning time. Default 1024 is the most common ActiveMQ production misconfiguration. Set 65536 or higher and alert at 70% usage.
• Alert on CLOSE_WAIT growth per port. It is the earliest indicator of the FD leak that eventually takes a connector down.
• Track certificate expiry as a metric, not a calendar reminder, for every SSL connector.
• Watch accept rate, not just connection count. A stable count can hide heavy churn; a sustained rate above 5x baseline is a reconnection storm in progress, and storms push connectors toward FD and thread limits.
• Add a canary per protocol. Send and receive a test message over each connector you expose. It is the only check that proves the full path: accept, handshake, auth, and dispatch.

How Netdata helps

• Per-port connection counts over time make the partial failure visible: one connector flatlines while the others carry traffic, which is the signature this whole article is about.
• File descriptor usage per process, correlated with the broker’s connection count, separates “too many clients” from “FD leak” before the connector starts refusing.
• Socket state breakdowns (including CLOSE_WAIT) surface the leak pattern that starves the accept path while live connection counts look normal.
• Log-based alerts on Could not accept connection and Too many open files give you the broker’s own account of why it refused, with the exception attached.
• JVM GC pause metrics alongside connection count show the sawtooth pattern that distinguishes a GC death spiral from a genuine connector fault, so you do not restart a connector for a heap problem.
• Alerting on “zero new connections for >30s on a serving connector” turns the playbook threshold into an automatic page instead of a user-reported incident.

Related guides

• ActiveMQ InactivityIOException: Channel was inactive for too long
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage
• ActiveMQ offline durable subscriber pending messages: the silent storage leak
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss
• ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms
• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill