NATS Authentication Timeout: clients that connect but never finish the handshake

Your NATS server log shows lines like this, and clients are complaining they cannot connect:

[ERR] 10.2.3.14:51824 - cid:1042 - Authentication Timeout

The confusing part: the TCP connection succeeded. The client reached the server. But the authentication handshake never finished, so the server sent -ERR 'Authentication Timeout' and closed the connection. This is not a bad-credentials problem. It is a timing problem: credentials (or the TLS handshake that must precede them) never arrived within the server’s auth window. For the related case where credentials arrive but are rejected, the log string is Authorization Violation and the diagnosis is different.

What this means

After a client establishes a TCP connection, the NATS server starts an auth timer. The client must complete any TLS upgrade and send a valid CONNECT protocol message before the timer fires. If it does not, the server writes -ERR 'Authentication Timeout' to the socket and closes it.

Key configuration facts:

• The timer is set by timeout inside the authorization block, in seconds (fractional values allowed, e.g. 3.5; duration strings like 10s are not accepted here).
• If timeout is unset or 0, the default is 1 second more than the TLS handshake timeout when TLS is configured, and 2 seconds otherwise.
• The TLS handshake timeout (tls { timeout }) defaults to 2 seconds in current server versions.
• So the effective default auth timeout is 3 seconds when TLS is enabled and 2 seconds without TLS.
• The auth timeout must be longer than the TLS timeout, because the TLS upgrade happens inside the auth window.

flowchart LR
  A[Client opens TCP] --> B{TLS enabled?}
  B -->|yes| C[TLS handshake
tls timeout default 2s]
  B -->|no| D[Server sends INFO]
  C --> D
  D --> E[Client sends CONNECT
with credentials]
  E --> F{Auth timer expired?}
  F -->|no| G[Connection established]
  F -->|yes| H["-ERR 'Authentication Timeout'"
connection closed]
  I[Slow resolver / hung client / LB stall] -.delays.-> E
  J[TLS stall or CPU saturation] -.delays.-> C

The consequence: anything that adds latency between “TCP accept” and “CONNECT received” eats into this budget. That includes the network path, TLS negotiation, server CPU contention, and any account resolution the server must do to validate the credentials.

Common causes

Quick checks

All read-only and safe to run during an incident.

# Count auth timeouts vs authorization violations in the server log
grep -c "Authentication Timeout" /var/log/nats/nats-server.log
grep -c "Authorization Violation" /var/log/nats/nats-server.log

# Check current connection state and churn
curl -s http://localhost:8222/varz | jq '{active: .connections, total: .total_connections, max: .max_connections}'

# Check server CPU and memory pressure (TLS handshakes are CPU-bound)
curl -s http://localhost:8222/varz | jq '{cpu_pct: .cpu, cores: .cores, mem: .mem}'

# Look at RTT for currently connected clients (worst first)
curl -s "http://localhost:8222/connz?sort=rtt&limit=10" | jq '.connections[] | {cid, name, ip, rtt}'

# Check server health (basic readiness; safe on JetStream servers too)
curl -s http://localhost:8222/healthz?js-server-only=true

Also confirm what the server is actually configured with. Look at the running config for the authorization block and the tls block. If authorization { timeout } is not set, the effective default is 1 second more than tls { timeout } when TLS is configured, or 2 seconds without TLS. If someone set the auth timeout shorter than the TLS timeout, every TLS client is racing a timer it cannot beat.

How to diagnose it

1. Confirm the failure class. Grep the log for both Authentication Timeout and Authorization Violation. Violations mean clients sent credentials and were rejected, which is a credentials problem, not this guide. Timeouts mean the credentials never arrived in time.

2. Characterize which clients time out. Note the source IPs in the log lines. All clients, or a subset? If a subset, what do they share: same subnet, same load balancer, same application, same WAN path? Timeouts limited to one LB backend point at the proxy. Timeouts from one application point at that client.

3. Check the timing budget. Read the effective auth timeout from config. With TLS at the 3-second effective default, a slow TLS handshake plus a slow CONNECT send can exhaust it. Cross-region paths with hundreds of milliseconds of RTT get very little slack, since the handshake involves multiple round trips.

4. Check server CPU during the bursts. curl -s http://localhost:8222/varz | jq .cpu. TLS handshakes are CPU-bound, and a reconnect storm can saturate cores so handshakes queue behind each other. If timeouts arrive in bursts right after a connection-count dip and spike, this is the connection-storm pattern. See NATS connection storm: reconnect thundering herd after a network event.

5. Check the account resolver (operator/JWT mode only). In decentralized auth setups, the server may need to fetch account JWTs from a resolver (URL resolver or NATS-based resolver) to validate a connecting client. If that lookup is slow or the resolver is unreachable, the auth window can expire before validation completes. Test resolver reachability and latency from the server itself. Timeouts that appear across many unrelated clients simultaneously, in a JWT deployment, are a strong resolver signal.

6. Bypass the middle. Have an affected client connect directly to a server’s IP and port, skipping any load balancer, ingress controller, or service mesh sidecar. If direct connections succeed while proxied ones time out, the middlebox is accepting TCP and forwarding late or stalling the handshake. L7 proxies and HTTP ingress controllers are not compatible with the NATS protocol anyway; only L4 passthrough works.

7. Check the client. On the client host, look for CPU saturation, GC pauses, or a blocked event loop that would delay sending CONNECT after the connection opens. A client under heavy load can open the socket and then not get scheduled in time to finish the handshake within a 2-3 second default window.

Metrics and signals to monitor

Fixes

Fix the timing configuration

If legitimate clients cannot finish the handshake inside the window, raise the auth timeout and, if TLS is involved, the TLS timeout with it. The auth timeout must stay strictly longer than the TLS timeout because the TLS upgrade happens inside the auth window:

tls {
  cert_file: "/etc/nats/tls/server.pem"
  key_file:  "/etc/nats/tls/server.key"
  timeout:   5
}

authorization {
  timeout: 10
}

Tradeoff: a longer auth timeout means half-open, unauthenticated connections hold server resources (file descriptors, per-connection buffers) for longer. On an internet-exposed listener this slightly raises the cost of slow-loris-style abuse. Raise it enough to cover your worst legitimate network path, not to infinity. Older server versions shipped with a 0.5 second TLS handshake default that was too low for cross-region deployments; current versions default to 2 seconds, which is still tight for high-latency paths. If you upgraded from an old config that pinned a small value, revisit it.

Fix the account resolver

If diagnosis points at the resolver: restore its availability first, then reduce dependence on its latency. A resolver that is down or slow turns every new connection into a timeout lottery. After fixing reachability, check whether the resolver timeout and caching behavior are appropriate for your deployment, and monitor resolver latency as a first-class signal.

Remove or fix the middlebox

If a load balancer or ingress is stalling the handshake, move NATS traffic to an L4 passthrough or have clients use a direct server list. NATS clients handle multiple server URLs and fail over themselves; a proxy in the middle adds latency, breaks the protocol if it is L7, and gives you exactly this failure mode.

Handle reconnect storms

If timeouts arrive in bursts after network events, the server is being overwhelmed by simultaneous handshakes. See NATS connection storm: reconnect thundering herd after a network event for the full playbook. Relevant here: current server versions support tls { connection_rate_limit } to cap new TLS connections per second, which smooths the CPU spike so handshakes complete instead of timing out in a queue.

Fix client-side behavior

For a hung or overloaded client, the fix is on the client host: CPU headroom, GC tuning, or an unblocked event loop. One client-library behavior worth knowing: in the Node.js client, Authentication Timeout is an authorization error. Two consecutive authentication failures abort reconnect by default, even when a reconnect policy is configured. For a server that has never been reached successfully, the client removes that server from its pool unless waitOnFirstConnect: true; that option does not disable the two-failure auth-abort behavior. Whether other client libraries share this behavior is not documented.

Prevention

• Set the auth timeout deliberately. Do not rely on the default unless your client paths are all low-latency. Compute the budget: TLS handshake time plus CONNECT round trip on your worst path, plus margin.
• Keep the ordering invariant. Auth timeout greater than TLS timeout, always. Enforce it in config review.
• Monitor churn, not just count. A stable connections number can hide constant connect-fail-retry cycles. Watch the total_connections delta. See NATS connection churn: a stable connection count hiding constant reconnects.
• Watch resolver health in JWT deployments. The resolver is part of the auth path. Its latency and availability belong on the same dashboard as server health.
• Keep NATS off L7 proxies. Direct server lists or L4 passthrough only.
• Alert on log rate, not existence. Isolated timeouts from a single misbehaving client are noise; a rising rate or a burst across many clients is signal.

How Netdata helps

• Connection churn correlation: Netdata tracks connections and total_connections from /varz per second, so the flat-count-but-high-churn signature of handshake failures is visible without log diving.
• CPU during connection bursts: per-second process CPU alongside connection-rate charts shows whether TLS handshake saturation coincides with timeout bursts.
• Stale and stalled connections: half-dead clients and write-path distress from the same network pathology surface next to connection counts, so you can see the blast radius of a bad path.
• Server health context: uptime and health-endpoint state let you rule out server restarts or JetStream recovery as the reason clients are reconnecting and timing out.
• Long retention for baselines: auth timeout events are often intermittent; per-second history makes it possible to line up a log burst with the network event, deploy, or resolver outage that caused it.

Related guides

• NATS JetStream AckWait tuning: matching the ack timeout to processing time
• NATS route RTT high: inter-server latency that triggers Raft elections
• NATS connection churn: a stable connection count hiding constant reconnects
• NATS connection storm: reconnect thundering herd after a network event
• NATS JetStream consumer lag growing: falling behind the stream
• NATS consumer stalled at MaxAckPending: delivery stops until messages are acked
• NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed
• NATS JetStream consumer stopped receiving messages: the diagnostic tree
• NATS context deadline exceeded: JetStream publish and request timeouts
• NATS crash loop: unexpected uptime resets and repeated restarts
• NATS file descriptor exhaustion: too many open files and the ulimit cliff
• NATS gateway disconnected: cross-cluster traffic cut in a supercluster