Your NATS server log shows lines like this, and clients are complaining they cannot connect:

[ERR] 10.2.3.14:51824 - cid:1042 - Authentication Timeout

The confusing part: the TCP connection succeeded. The client reached the server. But the authentication handshake never finished, so the server sent -ERR 'Authentication Timeout' and closed the connection. This is not a bad-credentials problem. It is a timing problem: credentials (or the TLS handshake that must precede them) never arrived within the server’s auth window. For the related case where credentials arrive but are rejected, the log string is Authorization Violation and the diagnosis is different.

What this means

After a client establishes a TCP connection, the NATS server starts an auth timer. The client must complete any TLS upgrade and send a valid CONNECT protocol message before the timer fires. If it does not, the server writes -ERR 'Authentication Timeout' to the socket and closes it.

Key configuration facts:

  • The timer is set by timeout inside the authorization block, in seconds (fractional values allowed, e.g. 3.5; duration strings like 10s are not accepted here).
  • If timeout is unset or 0, the default is 1 second more than the TLS handshake timeout.
  • The TLS handshake timeout (tls { timeout }) defaults to 2 seconds in current server versions.
  • So the effective default auth timeout is 3 seconds when TLS is enabled and about 1 second without TLS.
  • The auth timeout must be longer than the TLS timeout, because the TLS upgrade happens inside the auth window.
flowchart LR
  A[Client opens TCP] --> B{TLS enabled?}
  B -->|yes| C[TLS handshake
tls timeout default 2s]
  B -->|no| D[Server sends INFO]
  C --> D
  D --> E[Client sends CONNECT
with credentials]
  E --> F{Auth timer expired?}
  F -->|no| G[Connection established]
  F -->|yes| H["-ERR 'Authentication Timeout'"
connection closed]
  I[Slow resolver / hung client / LB stall] -.delays.-> E
  J[TLS stall or CPU saturation] -.delays.-> C

The consequence: anything that adds latency between “TCP accept” and “CONNECT received” eats into this budget. That includes the network path, TLS negotiation, server CPU contention, and any account resolution the server must do to validate the credentials.

Common causes

CauseWhat it looks likeFirst thing to check
Hung or slow clientTimeouts from one or a few client IPs; client process alive but not progressingIs the client process responsive? CPU, GC pauses, event loop stalls
Slow or unreachable account resolverTimeouts cluster in JWT/NKey (operator mode) deployments; bursts across many clients at onceResolver URL reachability from the server; resolver latency
TLS handshake stallTimeouts only on TLS listeners; may correlate with CPU spikes on the serverServer CPU (/varz cpu), TLS timeout vs auth timeout relationship
Load balancer or proxy in the pathLB accepts TCP but forwards late or buffers; timeouts affect clients behind that LB onlyConnect directly to a server, bypassing the LB, and compare
Reconnect storm overwhelming the serverBursts of timeouts right after a network event or server restart; total_connections climbing fastConnection churn and CPU during the burst
Network path latency mid-handshakeTimeouts mostly from remote/WAN clients, local clients fineRTT to affected clients via /connz

Quick checks

All read-only and safe to run during an incident.

# Count auth timeouts vs authorization violations in the server log
grep -c "Authentication Timeout" /var/log/nats/nats-server.log
grep -c "Authorization Violation" /var/log/nats/nats-server.log

# Check current connection state and churn
curl -s http://localhost:8222/varz | jq '{active: .connections, total: .total_connections, max: .max_connections}'

# Check server CPU and memory pressure (TLS handshakes are CPU-bound)
curl -s http://localhost:8222/varz | jq '{cpu_pct: .cpu, cores: .cores, mem: .mem}'

# Look at RTT for currently connected clients (worst first)
curl -s "http://localhost:8222/connz?sort=rtt&limit=10" | jq '.connections[] | {cid, name, ip, rtt}'

# Check server health (basic readiness; safe on JetStream servers too)
curl -s http://localhost:8222/healthz?js-server-only=true

Also confirm what the server is actually configured with. Look at the running config for the authorization block and the tls block. If authorization { timeout } is not set, compute the effective default: 1 second more than tls { timeout }, and the TLS timeout defaults to 2 seconds. If someone set the auth timeout shorter than the TLS timeout, every TLS client is racing a timer it cannot beat.

How to diagnose it

  1. Confirm the failure class. Grep the log for both Authentication Timeout and Authorization Violation. Violations mean clients sent credentials and were rejected, which is a credentials problem, not this guide. Timeouts mean the credentials never arrived in time.

  2. Characterize which clients time out. Note the source IPs in the log lines. All clients, or a subset? If a subset, what do they share: same subnet, same load balancer, same application, same WAN path? Timeouts limited to one LB backend point at the proxy. Timeouts from one application point at that client.

  3. Check the timing budget. Read the effective auth timeout from config. With TLS at the 3-second effective default, a slow TLS handshake plus a slow CONNECT send can exhaust it. Cross-region paths with hundreds of milliseconds of RTT get very little slack, since the handshake involves multiple round trips.

  4. Check server CPU during the bursts. curl -s http://localhost:8222/varz | jq .cpu. TLS handshakes are CPU-bound, and a reconnect storm can saturate cores so handshakes queue behind each other. If timeouts arrive in bursts right after a connection-count dip and spike, this is the connection-storm pattern. See NATS connection storm: reconnect thundering herd after a network event.

  5. Check the account resolver (operator/JWT mode only). In decentralized auth setups, the server may need to fetch account JWTs from a resolver (URL resolver or NATS-based resolver) to validate a connecting client. If that lookup is slow or the resolver is unreachable, the auth window can expire before validation completes. Test resolver reachability and latency from the server itself. Timeouts that appear across many unrelated clients simultaneously, in a JWT deployment, are a strong resolver signal.

  6. Bypass the middle. Have an affected client connect directly to a server’s IP and port, skipping any load balancer, ingress controller, or service mesh sidecar. If direct connections succeed while proxied ones time out, the middlebox is accepting TCP and forwarding late or stalling the handshake. L7 proxies and HTTP ingress controllers are not compatible with the NATS protocol anyway; only L4 passthrough works.

  7. Check the client. On the client host, look for CPU saturation, GC pauses, or a blocked event loop that would delay sending CONNECT after the connection opens. A client under heavy load can open the socket and then not get scheduled in time to finish the handshake within a 1-3 second window.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Auth timeout count in server logsThe direct symptom; rate matters more than existenceSustained rate above baseline, or bursts correlated with other events
total_connections delta vs connections (/varz)Churn: clients connecting and vanishing fast looks like a flat count but burns CPU and auth attemptsTotal climbing fast while active count is flat or flapping
Server cpu (/varz)TLS handshakes are the main CPU cost of connection setup; saturation delays handshakesCPU near saturation during timeout bursts
Client RTT (/connz rtt)High-RTT clients have less slack in a fixed auth windowRTT well above baseline for affected clients
stale_connections and stalled_clients (/varz)Half-dead or write-distressed connections often accompany the same network pathologyAny non-zero value sustained over minutes
Account resolver availability and latencyIn JWT/NKey mode, resolver slowness directly consumes the auth budgetResolver errors or latency approaching the auth timeout

Fixes

Fix the timing configuration

If legitimate clients cannot finish the handshake inside the window, raise the auth timeout and, if TLS is involved, the TLS timeout with it. The auth timeout must stay strictly longer than the TLS timeout because the TLS upgrade happens inside the auth window:

tls {
  cert_file: "/etc/nats/tls/server.pem"
  key_file:  "/etc/nats/tls/server.key"
  timeout:   5
}

authorization {
  timeout: 10
}

Tradeoff: a longer auth timeout means half-open, unauthenticated connections hold server resources (file descriptors, per-connection buffers) for longer. On an internet-exposed listener this slightly raises the cost of slow-loris-style abuse. Raise it enough to cover your worst legitimate network path, not to infinity. Older server versions shipped with a 0.5 second TLS handshake default that was too low for cross-region deployments; current versions default to 2 seconds, which is still tight for high-latency paths. If you upgraded from an old config that pinned a small value, revisit it.

Fix the account resolver

If diagnosis points at the resolver: restore its availability first, then reduce dependence on its latency. A resolver that is down or slow turns every new connection into a timeout lottery. After fixing reachability, check whether the resolver timeout and caching behavior are appropriate for your deployment, and monitor resolver latency as a first-class signal.

Remove or fix the middlebox

If a load balancer or ingress is stalling the handshake, move NATS traffic to an L4 passthrough or have clients use a direct server list. NATS clients handle multiple server URLs and fail over themselves; a proxy in the middle adds latency, breaks the protocol if it is L7, and gives you exactly this failure mode.

Handle reconnect storms

If timeouts arrive in bursts after network events, the server is being overwhelmed by simultaneous handshakes. See NATS connection storm: reconnect thundering herd after a network event for the full playbook. Relevant here: current server versions support tls { connection_rate_limit } to cap new TLS connections per second, which smooths the CPU spike so handshakes complete instead of timing out in a queue.

Fix client-side behavior

For a hung or overloaded client, the fix is on the client host: CPU headroom, GC tuning, or an unblocked event loop. One client-library quirk worth knowing: in the Node.js client, a connection closed with Authentication Timeout may not auto-reconnect even with unlimited reconnect attempts configured; setting waitOnFirstConnect: true changes that behavior. Whether the same quirk exists in other client libraries is not documented.

Prevention

  • Set the auth timeout deliberately. Do not rely on the default unless your client paths are all low-latency. Compute the budget: TLS handshake time plus CONNECT round trip on your worst path, plus margin.
  • Keep the ordering invariant. Auth timeout greater than TLS timeout, always. Enforce it in config review.
  • Monitor churn, not just count. A stable connections number can hide constant connect-fail-retry cycles. Watch the total_connections delta. See NATS connection churn: a stable connection count hiding constant reconnects.
  • Watch resolver health in JWT deployments. The resolver is part of the auth path. Its latency and availability belong on the same dashboard as server health.
  • Keep NATS off L7 proxies. Direct server lists or L4 passthrough only.
  • Alert on log rate, not existence. Isolated timeouts from a single misbehaving client are noise; a rising rate or a burst across many clients is signal.

How Netdata helps

  • Connection churn correlation: Netdata tracks connections and total_connections from /varz per second, so the flat-count-but-high-churn signature of handshake failures is visible without log diving.
  • CPU during connection bursts: per-second process CPU alongside connection-rate charts shows whether TLS handshake saturation coincides with timeout bursts.
  • Stale and stalled connections: half-dead clients and write-path distress from the same network pathology surface next to connection counts, so you can see the blast radius of a bad path.
  • Server health context: uptime and health-endpoint state let you rule out server restarts or JetStream recovery as the reason clients are reconnecting and timing out.
  • Long retention for baselines: auth timeout events are often intermittent; per-second history makes it possible to line up a log burst with the network event, deploy, or resolver outage that caused it.